Modeling method and system based on point cloud and vision algorithm coupling and related equipment

Abstract

The invention discloses a modeling method and system based on coupling of a point cloud and a visual algorithm, and related equipment, according to the scheme, point cloud data of a radar is combined with a visual image, geometric precision and texture details can be integrated, and a more complete three-dimensional model is formed. And meanwhile, the geometric accuracy of the point cloud and the high reduction degree of the image are combined and utilized, so that the precision and the visual effect of three-dimensional reconstruction are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of project engineering information technology, and in particular to a system and method for real-world reconstruction of an engineering project based on point cloud and visual algorithm coupling. Background Art

[0002] With the rapid development of the global construction industry, the complexity of engineering projects continues to increase. Traditional modeling and construction monitoring methods are difficult to meet the high-precision and high-efficiency requirements of modern engineering. In order to meet these challenges, the application of digital construction and intelligent technology has become particularly important. As a key technical means, 3D reconstruction technology can help project managers fully understand the actual situation of the construction site and monitor the progress of the project in real time. However, how to quickly and accurately generate high-quality 3D models in complex environments is still a difficult problem in the current technical field.

[0003] Traditional 3D modeling methods mainly rely on laser scanning (LiDAR) or photogrammetry technology. These methods have their own advantages and disadvantages: laser scanning technology can provide high-precision point cloud data, which is suitable for large-scale architectural or terrain reconstruction, but it is limited by the detail restoration of complex scenes and it is difficult to generate high-fidelity texture information. Photogrammetry technology relies on visual images and can generate delicate texture effects, but it has the problem of insufficient accuracy in depth and geometric structure reconstruction, especially in complex geometric scenes of buildings, which can easily lead to information loss or distortion. In addition, existing 3D reconstruction methods usually require manual operation and complex post-processing, and cannot achieve automated modeling and real-time updates. This makes it difficult to meet the needs of project managers for real-time monitoring and efficient decision-making in rapidly changing construction sites.

[0004] Therefore, the existing three-dimensional modeling methods have low accuracy due to the above-mentioned technical problems. It can be seen that how to improve the accuracy of three-dimensional modeling is a problem that needs to be solved in this field. Summary of the invention

[0005] In view of the technical problem that existing three-dimensional modeling methods have low accuracy, the purpose of the present invention is to provide a modeling method based on the coupling of point cloud and visual algorithm, which can automatically generate high-precision three-dimensional models, present complex building structures and details, and further improve the accuracy of three-dimensional modeling; on this basis, the present invention further provides a project reality reconstruction system and related equipment that can implement this method.

[0006] In order to achieve the above object, the present invention provides a modeling method based on the coupling of point cloud and visual algorithm, and the project reality reconstruction method comprises the following steps:

[0007] Step 1: Input and preprocessing of point cloud data:

[0008] Acquire point cloud data and pre-process the point cloud data;

[0009] Step 2: Gaussian distribution modeling:

[0010] Use the preprocessed point cloud data and transform it into a Gaussian distribution;

[0011] Step 3: Visibility and occlusion handling:

[0012] Using the viewpoint information, determine the visibility of each Gaussian distribution formed in step 2 above at different viewing angles, exclude the blocked parts, and simulate how the Gaussian distribution is projected onto the screen at different viewing angles through ray tracing technology to generate realistic images;

[0013] Step 4: Projection and Rendering:

[0014] Project the 3D Gaussian distribution onto the 2D image plane from different viewing angles and calculate the color and transparency so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image.

[0015] Step 5: Optimize and rebuild:

[0016] Compare the color difference of each pixel in the generated image and the real image, and gradually improve the reconstruction result by iteratively optimizing the parameters of the Gaussian kernel and reducing the error between the model and the actual image according to the loss function;

[0017] Step 6: Multi-view fusion output:

[0018] The rendering results from different perspectives are fused to generate the final 3D model output, ensuring the consistency of the model from all perspectives.

[0019] Furthermore, in step 1, the point cloud data obtained is further optimized by a cleaning method of statistical filtering, radius filtering and voxel grid filtering.

[0020] Furthermore, in step 2, the three-dimensional spatial coordinates of the preprocessed point cloud data are extracted, and a covariance matrix is ​​assigned to the point to describe the local influence range of the point.

[0021] Furthermore, the viewpoint information in step 3 is obtained by collecting multi-view information of the front-end acquisition device, as well as the direction of the view and the projection matrix.

[0022] Furthermore, in step 3, the center point of each Gaussian distribution in the three-dimensional point cloud needs to be transformed by a projection matrix to calculate its projection coordinates on the two-dimensional image plane.

[0023] Furthermore, the visibility is judged based on the projection result to determine whether the depth value of the point is within the visible range of the camera. If a Gaussian distribution point is behind the viewpoint or beyond the field of view, it is marked as invisible.

[0024] Furthermore, in step 3, the ray tracing and occlusion calculation are accelerated by using a hierarchical bounding volume space segmentation algorithm, and the occluded Gaussian kernel is gradually hidden by appropriately adjusting the transparency parameter of the Gaussian distribution to enhance the visual smoothness of the image.

[0025] Furthermore, the three-dimensional model after fusion in step 5 needs to optimize the parameters of the Gaussian kernel and adjust the surface smoothness of the local area to ensure that the model retains sufficient details while avoiding noise and overfitting.

[0026] In order to achieve the above object, the present invention provides a modeling system based on point cloud and visual algorithm coupling, comprising:

[0027] A point cloud data input and preprocessing module, which acquires the point cloud data and performs filtering and noise reduction processing on the point cloud data to further optimize the data;

[0028] A Gaussian distribution modeling module, which interacts with the input of point cloud data and data of the preprocessing module to convert the preprocessed point cloud data into a Gaussian distribution;

[0029] The projection and rendering module interacts with the Gaussian distribution modeling module data, projects the three-dimensional Gaussian distribution from different perspectives onto the two-dimensional image plane, and calculates the color and transparency, so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image.

[0030] The optimization and reconstruction module interacts with the projection and rendering module data, compares the color difference of each pixel in the generated image and the real image, iteratively optimizes the parameters of the Gaussian kernel, reduces the error between the model and the actual image according to the loss function, and gradually improves the reconstruction result.

[0031] A multi-view fusion output module interacts with the optimization and reconstruction module data to fuse the rendering results from different viewpoints to generate a final 3D model output, thereby ensuring the consistency of the model under each viewpoint and achieving high-quality reconstruction effects.

[0032] In order to achieve the above-mentioned object, the present invention provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the modeling method based on the coupling of point cloud and visual algorithm.

[0033] The modeling method provided by the present invention combines the point cloud data of the radar with the visual image, thereby achieving both geometric accuracy and texture details to form a more complete three-dimensional model.

[0034] At the same time, the combination of the geometric accuracy of point clouds and the high degree of image restoration greatly improves the accuracy and visual effects of three-dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0036] Figure 1 This is a flow chart of the modeling method based on the coupling of point cloud and visual algorithm;

[0037] Figure 2 This is the effect diagram of the gradual reconstruction of the model and the actual image based on the modeling method coupled with point cloud and visual algorithm. DETAILED DESCRIPTION

[0038] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below with reference to specific diagrams.

[0039] In order to solve the technical problem of low precision in existing three-dimensional modeling methods, the present invention combines radar point cloud data with visual images to achieve both geometric precision and texture details to form a more complete three-dimensional model.

[0040] At the same time, the combination of the geometric accuracy of point clouds and the high degree of image restoration greatly improves the accuracy and visual effects of three-dimensional reconstruction.

[0041] The project reality reconstruction method provided by the present invention comprises the following steps:

[0042] Step 1: Input and preprocessing of point cloud data:

[0043] First, point cloud data needs to be acquired. Point cloud data acquisition is the basis of 3D reconstruction. Point cloud data can be acquired in different ways according to the scene scale, object shape and scanning effect requirements. For example, point cloud data can be acquired in the following three ways:

[0044] 1. Use a multi-view camera to capture images from multiple angles, calculate depth information from them to reconstruct the three-dimensional shape of the object, which is suitable for the reconstruction of large-scale scenes;

[0045] 2. Directly obtain scene depth information through depth sensors such as Kinect or LiDAR, which is suitable for small objects or scenes with rich details;

[0046] 3. Using structured light scanning, the 3D shape of the object surface is calculated by projecting a grating pattern and analyzing the deformation, which is suitable for high-precision 3D scanning.

[0047] Furthermore, the point cloud data obtained according to the above method is usually sparse and incomplete, and may be accompanied by measurement noise and occlusion. Therefore, data preprocessing is required to ensure the accuracy of the subsequent data. Data preprocessing is the basis for the coupling of the subsequent point cloud and visual algorithm.

[0048] Specifically, the point cloud data needs to be filtered and denoised at the most basic level, and the data can be further optimized through cleaning methods such as statistical filtering, radius filtering, and voxel grid filtering.

[0049] Among them, statistical filtering removes isolated points and outliers by calculating the number of neighbors of each point; radius filtering sets a fixed radius range to remove isolated points beyond the range; voxel grid filtering reduces the amount of data and retains key geometric structures by dividing the point cloud into grids and merging points within the same voxel.

[0050] Then, the point cloud data from different sensors or perspectives are mapped into a unified coordinate system through coordinate normalization to ensure the consistency of subsequent processing.

[0051] Specifically, the point cloud data is centrally translated to align it with the coordinate origin, and the scale is normalized to scale the point cloud to the unit cube to eliminate the impact of size differences of different objects. For point cloud data from multiple perspectives, it is necessary to extract key feature points from images of different perspectives through the SIFT feature matching algorithm, and match the correspondence between them to provide rotation and translation estimates for the preliminary alignment of the point cloud. Then use the RANSAC random sampling consistency algorithm to screen reliable feature point pairs, and calculate a rough transformation matrix to complete the data alignment to ensure that the point cloud data is aligned in the same coordinate system. Then use the ICP (Iterative Closest Point) algorithm to iteratively adjust the rotation and translation matrices of the point cloud to minimize the distance between the corresponding points of the two point clouds; for overly dense data sets, the amount of data can be reduced by downsampling to improve computational efficiency.

[0052] Step 2: Gaussian distribution modeling

[0053] This step is based on the point cloud data preprocessed in step 1, and converts the preprocessed point cloud data into a Gaussian distribution (normal distribution).

[0054] Specifically, for each point in the processed point cloud, its three-dimensional spatial coordinates, (x, y, z), are extracted, and these points are regarded as the center of the Gaussian distribution, that is, the mean μ = (x, y, z).

[0055] Then, a covariance matrix Σ is assigned to each point to describe the local influence range of the point. The covariance matrix is ​​usually a diagonal matrix, which represents the spatial distribution width of the point in all directions. It balances the smoothness and detail retention of the model by controlling the degree of expansion in different directions in space, copes with data sparsity and noise, improves the consistency of multi-view fusion, and accurately calculates the contribution of each point to the pixel color during rendering.

[0056] Next, determine the covariance matrix. For each point, select a fixed-size neighborhood and select several nearest points from the neighbors of the point based on the Euclidean distance. Calculate the distance vector Δx between these neighborhood points and the center point i =x i -μ, and based on these difference vectors, calculate the elements of the covariance matrix.

[0057] The covariance matrix Σ is estimated by the following formula:

[0058]

[0059] N is the number of points in the neighborhood. The matrix describes how much a point is expanded or compressed in different directions, i.e. how it affects the surrounding area.

[0060] Step 3: Visibility and Occlusion

[0061] Using the viewpoint information, the visibility of each Gaussian distribution formed in step 2 above under different viewing angles is determined, excluding the obscured parts. Ray tracing technology is used to simulate how the Gaussian distribution is projected onto the screen under different viewing angles, thereby generating realistic images.

[0062] Specifically, the viewpoint information is obtained by collecting multi-view information of the camera or sensor, that is, the position information of the viewpoint (x v ,y v ,z v ), as well as the direction of the viewing angle and the projection matrix, which determine the visibility of each 3D point cloud from different viewing angles.

[0063] The projection matrix here can accurately obtain the intrinsic parameter matrix and the extrinsic parameter matrix through the calibration process, so as to construct a complete projection matrix and accurately map the multi-view point cloud data and viewpoint information into a unified coordinate system.

[0064] Then, the center point μ = (x, y, z) of each Gaussian distribution in the three-dimensional point cloud is transformed by the projection matrix to calculate its projection coordinates on the two-dimensional image plane. The projection transformation can be expressed as:

[0065] p′=P·μ

[0066] Where P is the projection matrix and p′ is the projection coordinate of the point. This step provides the basis for subsequent visibility judgment and occlusion processing.

[0067] The visibility judgment is to calculate whether each Gaussian kernel is visible in the current viewing angle based on its relative position to the camera or sensor. If a Gaussian distribution point is behind the viewpoint or beyond the field of view, it is marked as invisible.

[0068] The specific implementation method is to determine whether the depth value (z value) of the point is within the visible range of the camera based on the projection result. If the z value is too large (i.e. too far from the camera) or negative (behind the viewpoint), the point is considered invisible. Use depth buffer technology to record the depth value of each projected pixel. When rendering, only the pixels closest to the camera are retained, so that the parts blocked by other Gaussian kernels can be excluded.

[0069] Ray tracing and occlusion processing are then performed, and ray tracing technology is used to handle visibility and occlusion problems. Each ray is emitted from the camera and passes through the three-dimensional point cloud scanned in the scene. When the ray encounters a Gaussian kernel obtained after preprocessing the point cloud data, the color, transparency, and depth information of the kernel are recorded. This process allows simulation of how light interacts with the Gaussian kernel to generate realistic occlusion effects. For each ray, determine which Gaussian kernels it intersects with, and determine the color and transparency of each pixel by calculating the overlapping area of ​​the ray and the Gaussian distribution. The formula is as follows:

[0070]

[0071] Where Ti is the transparency of each Gaussian kernel, Ci is the color information of the Gaussian kernel, and C is the final synthesized pixel color.

[0072] Finally, occlusion optimization is performed, and occlusion processing is repeated under multiple viewing angles. Ray tracing is used to ensure that the visibility of each Gaussian kernel in each viewing angle is correctly processed. A hierarchical bounding volume space segmentation algorithm is used to accelerate ray tracing and occlusion calculations. By appropriately adjusting the transparency parameters of the Gaussian distribution, the occluded Gaussian kernel is gradually hidden to enhance the visual smoothness of the image.

[0073] The hierarchical bounding volume space segmentation algorithm organizes the three-dimensional Gaussian kernels in the scene into a tree structure, so that each node represents a bounding volume, which contains multiple Gaussian kernels.

[0074] When ray tracing starts, first check whether the ray intersects with the bounding volume of the root node. If not, no further testing is required for the node and its child nodes. If the ray intersects with the bounding volume of the root node, recursively check its child nodes until the bottom-level bounding volume is reached.

[0075] The above steps generate realistic images that conform to actual physical laws.

[0076] Step 4: Projection and Rendering:

[0077] Furthermore, the three-dimensional Gaussian distribution is projected onto the two-dimensional image plane from different perspectives, and the color and transparency are calculated, so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image.

[0078] Gaussian distribution projection to the two-dimensional image plane is to project the Gaussian distribution kernel in the three-dimensional space to the two-dimensional image plane from each viewpoint. First, the center coordinates μ = (x, y, z) of each Gaussian distribution kernel are transformed by the viewpoint projection matrix P to obtain its coordinates p' = (x', y') on the image plane. The projection calculation is as follows:

[0079] p′=P·μ

[0080] Here, P represents the projection matrix of the camera, which transforms 3D points into 2D coordinates under the viewpoint.

[0081] After that, the Gaussian kernel is expanded in the two-dimensional plane. Each Gaussian distribution will form a blurred circular or elliptical area on the two-dimensional plane after projection. Its shape and size are determined by the covariance matrix Σ of the Gaussian kernel. By calculating the transformation of the covariance matrix on the projection plane, the shape of the two-dimensional Gaussian distribution can be determined. For each pixel point, calculate its distance from the center of the projected Gaussian kernel. The closer the pixel is to the center, the greater the weight, indicating that the color influence is stronger. According to the formula of the Gaussian distribution function:

[0082]

[0083] For each pixel, it may be affected by multiple Gaussian kernels. In order to calculate the final color C and transparency α of the pixel, the color and transparency of all Gaussian kernels need to be weighted averaged. Assuming that a pixel is covered by N Gaussian kernels, the color and transparency are calculated as follows:

[0084]

[0085] Among them, Ci is the color of the i-th Gaussian kernel, αi is the transparency of the i-th Gaussian kernel, and Gi(p′) is the weight of the Gaussian kernel at the pixel point. This weighted average method ensures a smooth transition between color and transparency and avoids abrupt color differences or transparency jumps in pixels. At each pixel, the color information of multiple Gaussian kernels is weighted and superimposed to generate the final color.

[0086] In the actual rendering process, in order to avoid the jagged effect caused by pixel boundaries, supersampling technology is used. When calculating the color value of each pixel, the sub-pixels of the pixel are first projected and weighted, and finally the results of these sub-pixels are averaged to generate a smoother rendering effect.

[0087] Through the above steps, the Gaussian distribution kernel in the three-dimensional space can be accurately projected onto the two-dimensional image, and the color and transparency of each pixel can be calculated by weighted averaging to generate a visually coherent, smooth and detail-rich image.

[0088] Step 5: Optimize and rebuild:

[0089] Furthermore, by iteratively optimizing the parameters of the Gaussian kernel (such as position and variance), the error between the model and the actual image is reduced according to the loss function, and the reconstruction result is gradually improved.

[0090] In order to make the generated 3D model match the target image, we first need to define a loss function to measure the difference between the projected image and the target image. Based on the color error loss Lcolor, it is calculated by comparing the color difference of each pixel in the generated image and the real image. The loss function uses the mean square error at the pixel level, and the formula is as follows:

[0091]

[0092] Among them, Irendered(pi) is the color value of the i-th pixel in the generated image,

[0093] Itarget(pi) is the color value of the corresponding pixel in the target image, and N is the total number of pixels.

[0094] In order to avoid overfitting or overly aggressive parameter adjustments, regularization loss is introduced to prevent the variance of the Gaussian distribution from being too large or the position from being too offset. Regularization constraints are applied to the variance of the Gaussian kernel:

[0095]

[0096] Among them, λ is the weight factor, tr(Σi) is the trace of the i-th Gaussian kernel covariance matrix, that is, its diffusion range.

[0097] In order to optimize the Gaussian distribution, its position, variance and color parameters need to be initialized first. The position is obtained from the initial three-dimensional point cloud data, and the position of each point cloud is used as the initial center coordinate of the Gaussian kernel. The initial variance Σ is set to a smaller value, preferably between 0.01 and 0.05, to ensure that the influence range of the Gaussian kernel is concentrated in the local area of ​​the point cloud and can accurately describe the detail features to ensure that the initial model is fine enough. The color of each Gaussian kernel is initialized by sampling the color data of the point cloud or the initial image data. For point clouds without clear color data, the average color of the multi-view image is used as the initial value.

[0098] Then, the gradient descent method is used to gradually adjust the parameters of the Gaussian distribution. First, the gradient of the loss function is calculated to determine the adjustment direction and amplitude of each parameter in the current step. The gradient calculation formula for position μ, variance Σ and color C is as follows:

[0099]

[0100] Among them, η is the learning rate and L is the total loss function. The gradient of the loss function is calculated through each iteration, and the parameters are updated so that the projected image gradually approaches the target image.

[0101] In each optimization iteration, the system compares the projection results at different views with their respective target images and calculates the total loss of multiple views based on the following companies:

[0102]

[0103] Among them, M is the number of perspectives, and each perspective has an independent loss calculation. By optimizing the projection errors of all perspectives at the same time, it is ensured that the generated 3D model can accurately restore the real scene from each perspective. In multi-perspective optimization, some perspectives may contribute more to the final reconstruction effect, while other perspectives contribute less. The mean square error between each perspective and the real image can be calculated to quantify the reconstruction error. The error of each perspective is compared with the maximum error to generate a relative weight. A dynamic threshold is set. When the error of a certain perspective is lower than the threshold, its weight can be increased. By dynamically adjusting the weight of each perspective in the total loss according to the error size of each perspective, the participation order of the perspective is adjusted, so that the optimization algorithm can give priority to the perspectives that contribute more to the reconstruction effect, accelerate convergence and improve the overall reconstruction accuracy.

[0104] During the optimization process, when the value of the loss function L gradually decreases and tends to be stable, it means that the parameters of the Gaussian distribution have achieved a good optimization effect. A threshold ∈ can be set, and the optimization is stopped when the change of the loss function is less than the threshold.

[0105] Through the above steps, the loss function gradually guides the adjustment of Gaussian distribution parameters, making the projected image closer and closer to the real image, and finally achieving a high-precision 3D reconstruction model. For specific effect diagrams, see Figure 2 ,.

[0106] Step 6: Multi-view fusion output:

[0107] The rendering results from different perspectives are fused to generate the final 3D model output, ensuring the consistency of the model from each perspective and achieving high-quality reconstruction effects.

[0108] First, the image or point cloud data obtained from different perspectives are preprocessed. The image data from these perspectives needs to be combined with the Gaussian distribution in the 3D scene to unify them into a global coordinate system. To ensure the consistency of the perspective data, the system uses the camera parameters of each perspective to align the point cloud or image projection of each perspective.

[0109] The fusion of Gaussian distribution needs to consider the feature overlap area of ​​each perspective. For different perspectives of the same object surface, the parameters of Gaussian distribution (position, variance, color, etc.) need to be fused with each other.

[0110] First, the position parameters are adjusted by taking a weighted average of the Gaussian kernel centers of the same object at different viewing angles. The weights can be calculated based on the visibility or error size of each viewing angle, so that Gaussian distributions from more viewing angles have a greater impact on the final position.

[0111] Variance fusion is performed by considering the uncertainty at different viewing angles. Smaller variances indicate more accurate modeling of the object surface, while larger variances indicate blurred areas. Therefore, a smaller variance value is selected during fusion to make the model surface more refined. Color fusion is accomplished by weighted averaging the color information at different viewing angles, ensuring that the color information remains consistent between viewing angles.

[0112] For areas with large color differences, the system can adjust the weights according to the brightness and color differences at different viewing angles to reduce color mutations. For the fusion of different viewing angles, the occlusion problem needs to be considered. The visibility of each Gaussian kernel may be different at each viewing angle, and part of the Gaussian distribution may be blocked by other objects, so the system needs to calculate the occluded areas and assign lower weights to these areas. In this way, the negative impact of the occluded areas on the fusion results is avoided.

[0113] The fused 3D Gaussian distribution will be unified into a complete global 3D model. When generating the unified model, the system will smooth the global model based on the fusion results of multiple perspectives to avoid detail faults or mutations in the model. By resampling and interpolating the fused Gaussian distribution, a unified and continuous 3D surface is generated.

[0114] In order to further improve the accuracy of the reconstructed model, the system will refine the fused 3D model. The specific operations include optimizing the parameters of the Gaussian kernel and adjusting the surface smoothness of the local area to ensure that the model retains enough details while avoiding noise and overfitting. By adjusting the variance of the Gaussian distribution of each area, the details can be further enhanced in the local area.

[0115] Through multi-perspective Gaussian distribution fusion and rendering optimization, a three-dimensional model that highly restores the real object or scene is finally generated.

[0116] The method for project reality reconstruction provided by the present invention first acquires and pre-processes point cloud data;

[0117] Next, the pre-processed point data is Gaussian distributed, and the visibility of each Gaussian distribution formed at different viewing angles is determined, excluding the obscured parts. And by ray tracing technology, how the Gaussian distribution is projected onto the screen at different viewing angles can be simulated to generate realistic images.

[0118] Secondly, the three-dimensional Gaussian distribution is projected onto the two-dimensional image plane from different perspectives, and the color and transparency are calculated, so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image.

[0119] Secondly, by iteratively optimizing the parameters of the Gaussian kernel (such as position and variance), the error between the model and the actual image is reduced according to the loss function, and the reconstruction result is gradually improved.

[0120] Finally, the rendering results from different perspectives are fused to generate the final 3D model output, ensuring the consistency of the model from each perspective and achieving high-quality reconstruction effects.

[0121] The modeling method based on the coupling of point cloud and visual algorithm given in this example solution can constitute a corresponding software program and form a corresponding modeling system when it is applied. When the software program is running, it will execute the above modeling method and store it in the corresponding storage medium for the processor to call and execute.

[0122] The modeling system based on the coupling of point cloud and visual algorithm thus formed mainly includes the following functions: point cloud data input and preprocessing module, Gaussian distribution modeling module, visibility and occlusion processing module, projection and rendering module, optimization and reconstruction module and multi-view fusion output module.

[0123] The point cloud data input and preprocessing module acquires the point cloud data and performs filtering and noise reduction to further optimize the data.

[0124] The point cloud data input and preprocessing module can be configured to execute the above-mentioned point cloud data input and preprocessing steps to realize corresponding functions.

[0125] The Gaussian distribution modeling module in this system interacts with the point cloud data input and preprocessing module data to convert the preprocessed point cloud data into Gaussian distribution (normal distribution).

[0126] The Gaussian distribution modeling module can be configured to execute the above-mentioned Gaussian distribution modeling steps to achieve corresponding functions.

[0127] The projection and rendering module in this system interacts with the Gaussian distribution modeling module to project the three-dimensional Gaussian distribution onto the two-dimensional image plane from different perspectives, and calculates the color and transparency, so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image.

[0128] The projection and rendering module can be configured to execute the above-mentioned projection and rendering steps to realize corresponding functions.

[0129] In order to avoid the jagged effect caused by pixel boundaries, the projection and rendering module uses supersampling technology to calculate the color value of each pixel. The sub-pixels of the pixel are projected and weighted, and the results of these sub-pixels are finally averaged to generate a smoother rendering effect.

[0130] The optimization and reconstruction module in this system interacts with the projection and rendering module data, compares the color difference of each pixel in the generated image and the real image, it iteratively optimizes the parameters of the Gaussian kernel (such as position and variance), reduces the error between the model and the actual image according to the loss function, and gradually improves the reconstruction result.

[0131] The optimization and reconstruction module can be configured to execute the above optimization and reconstruction steps to achieve corresponding functions.

[0132] The multi-view fusion output module in this system interacts with the optimization and reconstruction module data to fuse the rendering results from different viewpoints to generate the final 3D model output, ensuring the consistency of the model under each viewpoint and achieving high-quality reconstruction effects.

[0133] The multi-view fusion output module can be configured to execute the above multi-view fusion output steps to achieve corresponding functions.

[0134] Based on the above-mentioned modeling method based on coupling of point cloud and visual algorithm, an embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon, which implements the steps of the above-mentioned method for real-world reconstruction of engineering projects when executed by a processor.

[0135] An embodiment of the present invention further provides a processor, which is used to run a program, wherein the program executes the steps of the engineering project reality reconstruction method when running.

[0136] An embodiment of the present invention also provides a terminal device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program code is loaded and executed by the processor to implement the steps of the above-mentioned engineering project reality reconstruction method.

[0137] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the above-mentioned engineering project reality reconstruction method.

[0138] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0145] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0146] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0147] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0148] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0149] The above-mentioned method of the present invention, or a specific system unit, or a part of the same, is a pure software architecture, which can be arranged on a physical medium, such as a hard disk, an optical disk, or any electronic device (such as a smart phone, a computer-readable storage medium) through program code. When the machine loads the program code and executes it (such as a smart phone loading and executing it), the machine becomes a device for implementing the present invention. The above-mentioned method and device of the present invention can also be transmitted in the form of program code through some transmission media, such as cables, optical fibers, or any transmission mode. When the program code is received, loaded and executed by a machine (such as a smart phone), the machine becomes a device for implementing the present invention.

[0150] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A modeling method based on point cloud and visual algorithm coupling, characterized in that: The project reality reconstruction method comprises the following steps: Step 1: Input and preprocessing of point cloud data: Acquire point cloud data and pre-process the point cloud data; Step 2: Gaussian distribution modeling: Use the preprocessed point cloud data and transform it into a Gaussian distribution; Step 3: Visibility and occlusion handling: Using the viewpoint information, determine the visibility of each Gaussian distribution formed in step 2 above at different viewing angles, exclude the blocked parts, and simulate how the Gaussian distribution is projected onto the screen at different viewing angles through ray tracing technology to generate realistic images; Step 4: Projection and Rendering: Project the 3D Gaussian distribution onto the 2D image plane from different viewing angles and calculate the color and transparency so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image. Step 5: Optimize and rebuild: Compare the color difference of each pixel in the generated image and the real image, and gradually improve the reconstruction result by iteratively optimizing the parameters of the Gaussian kernel and reducing the error between the model and the actual image according to the loss function; Step 6: Multi-view fusion output: The rendering results from different perspectives are fused to generate the final 3D model output, ensuring the consistency of the model from all perspectives.

2. The modeling method based on point cloud and visual algorithm coupling according to claim 1 is characterized in that: In the step 1, the acquired point cloud data is further optimized by means of a cleaning method including statistical filtering, radius filtering and voxel grid filtering.

3. The modeling method based on point cloud and visual algorithm coupling according to claim 1 is characterized in that: In step 2, the three-dimensional spatial coordinates of the preprocessed point cloud data are extracted, and a covariance matrix is ​​assigned to the point to describe the local influence range of the point.

4. The modeling method based on point cloud and visual algorithm coupling according to claim 1 is characterized in that: The acquisition of the viewpoint information in step 3 is performed by collecting multi-view information of the front-end acquisition device, as well as the direction of the view and the projection matrix.

5. The modeling method based on point cloud and visual algorithm coupling according to claim 1 is characterized in that: In step 3, the center point of each Gaussian distribution in the three-dimensional point cloud needs to be transformed by a projection matrix to calculate its projection coordinates on the two-dimensional image plane.

6. The modeling method based on point cloud and visual algorithm coupling according to claim 5 is characterized in that: Visibility judgment is based on the projection result to determine whether the depth value of the point is within the visible range of the camera. If a Gaussian distribution point is behind the viewpoint or beyond the field of view, it is marked as invisible.

7. The modeling method based on point cloud and visual algorithm coupling according to claim 6 is characterized in that: In step 3, the ray tracing and occlusion calculation are accelerated by using the hierarchical bounding volume space segmentation algorithm, and the occluded Gaussian kernel is gradually hidden by appropriately adjusting the transparency parameter of the Gaussian distribution to enhance the visual smoothness of the image.

8. The modeling method based on point cloud and visual algorithm coupling according to claim 1 is characterized in that: The fused three-dimensional model in step 5 also needs to optimize the parameters of the Gaussian kernel and adjust the surface smoothness of the local area to ensure that the model retains sufficient details while avoiding noise and overfitting.

9. A modeling system based on point cloud and visual algorithm coupling, characterized in that: include: A point cloud data input and preprocessing module, which acquires the point cloud data and performs filtering and noise reduction processing on the point cloud data to further optimize the data; A Gaussian distribution modeling module, which interacts with the input of point cloud data and data of the preprocessing module to convert the preprocessed point cloud data into a Gaussian distribution; The projection and rendering module interacts with the Gaussian distribution modeling module data, projects the three-dimensional Gaussian distribution from different perspectives onto the two-dimensional image plane, and calculates the color and transparency, so that each pixel presents a delicate light and shadow effect, providing visual consistency for the final output image. The optimization and reconstruction module interacts with the projection and rendering module data, compares the color difference of each pixel in the generated image and the real image, iteratively optimizes the parameters of the Gaussian kernel, reduces the error between the model and the actual image according to the loss function, and gradually improves the reconstruction result. A multi-view fusion output module interacts with the optimization and reconstruction module data to fuse the rendering results from different viewpoints to generate a final 3D model output, thereby ensuring the consistency of the model under each viewpoint and achieving high-quality reconstruction effects.

10. A computer program product, characterized in that When executed on a data processing device, it is suitable for executing the steps of the modeling method based on the coupling of point cloud and visual algorithm as described in any one of claims 1 to 8.

Images