A denoising method, system and device for a 3D GS model of a target object

Abstract

The application provides a denoising method, system and device for a 3D GS model of a target object, the method comprising: obtaining original images of the target object from multiple perspectives and performing preprocessing to obtain a set of preprocessed images; performing sparse reconstruction processing on the set of preprocessed images to obtain a sparse point cloud data set; performing dense reconstruction processing on the sparse point cloud data set to obtain a dense point cloud data set; performing scale recovery and posture correction processing on the dense point cloud data set and performing filtering processing based on HSV color to obtain a filtered dense point cloud data set; performing Gaussian optimization and dense reconstruction processing on the sparse point cloud data set to obtain a Gaussian dense point cloud data set, combining 3D GS reconstruction technology to obtain a 3D GS model, and further obtaining a 3D GS model point cloud data set; calculating the distance of points in the filtered dense point cloud data set and the 3D GS model point cloud data set, removing points in the 3D GS model with a distance greater than a preset distance threshold, and obtaining a denoised 3D GS model. The 3D GS model obtained by the application can effectively remove noise.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, three-dimensional reconstruction and image processing, and in particular to a method, system and device for denoising a 3D Gaussian Splatting (3DGS) model of a target object. BACKGROUND

[0002] With the rapid development of three-dimensional reconstruction technology, three-dimensional models have been widely applied in various fields such as industrial detection, virtual reality, film production, medical imaging, etc. In these applications, generating high-quality and high-fidelity three-dimensional models is crucial for improving product reliability and enhancing user experience. However, existing three-dimensional reconstruction techniques often produce a large amount of noise when dealing with complex objects or limited shooting conditions, resulting in three-dimensional models that cannot meet the actual requirements in terms of quality.

[0003] Traditional three-dimensional reconstruction methods such as structured light reconstruction, laser slam, and image dense reconstruction can generate relatively accurate three-dimensional models in certain specific environments. However, these methods have high requirements for hardware devices and lighting conditions, and often cannot achieve high-quality reconstruction results in complex scenes or low-light environments. In addition, existing denoising techniques mostly focus on a single dimension, such as point cloud filtering or image processing-based denoising methods. These methods often fail to effectively remove noise and easily lose detailed information when dealing with complex three-dimensional models, resulting in unstable quality of the final three-dimensional models.

[0004] In recent years, 3D Gaussian Splatting (3DGS) technology has made significant progress in the field of three-dimensional reconstruction as a new three-dimensional reconstruction method. By modeling the target object with a Gaussian distribution and combining optimization algorithms to fine-tune the three-dimensional model, 3DGS can effectively reduce noise while preserving the details of the object's surface. This makes 3DGS technology capable of generating high-fidelity three-dimensional models while better handling complex objects and eliminating noise, thus meeting the demand for high-quality reconstruction.

[0005] However, although the 3DGS technology has significant limitations in denoising ability in complex scenes: (1) 3DGS relies on spherical harmonics to model lighting and material, which causes high-frequency noise (such as geometric spikes, isolated noise points) to be easily coupled into low-frequency signals, making it difficult for traditional filtering methods to effectively separate them; (2) the geometric optimization of the Gaussian sphere is highly dependent on initial parameter assumptions, which can easily introduce local bias and destroy geometric consistency in occlusion, weak texture, or dynamic lighting scenes; (3) existing denoising methods focus on a single modality (such as point cloud or image filtering), making it difficult to balance the conflict between noise removal and detail preservation, and lacking a cross-modal collaboration mechanism. In addition, traditional 3DGS technology does not fully utilize the prior denoising ability of mature three-dimensional dense models, resulting in limited reconstruction quality due to the inherent limitations of a single model, making it difficult to generate high-quality 3DGS models. SUMMARY

[0006] The present application is directed to the shortcomings in the prior art, and provides a target object 3DGS model denoising method, system and device.

[0007] To solve the above technical problems, the present application is solved by the following technical scheme:

[0008] A target object 3DGS model denoising method, comprising the following steps:

[0009] Obtain the original images of the target object from multiple perspectives to obtain an original image set and perform preprocessing to obtain a preprocessed image set, wherein all images in the original image set form a complete image of the target object;

[0010] Perform sparse reconstruction processing on the preprocessed image set to obtain a sparse point cloud data set;

[0011] Perform dense reconstruction processing on the sparse point cloud data set to obtain a dense point cloud data set;

[0012] Perform scale recovery and pose correction processing on the dense point cloud data set to obtain a corrected dense point cloud data set; perform filtering processing on the corrected dense point cloud data set based on an HSV color space threshold to obtain a filtered dense point cloud data set;

[0013] Perform Gaussian optimization and dense reconstruction processing on the sparse point cloud data set to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, combine 3DGS reconstruction technology to obtain a 3DGS model, and then obtain a 3DGS model point cloud data set;

[0014] Calculate the distance between corresponding three-dimensional points in the filtered dense point cloud data set and the 3DGS model point cloud data set, remove three-dimensional points in the 3DGS model whose distance is greater than a preset distance threshold, and obtain a denoised 3DGS model.

[0015] As an implementable manner, the original images of the target object from multiple perspectives are acquired to obtain an original image set and are preprocessed to obtain a preprocessed image set, including the following steps:

[0016] Based on the image acquisition device, surface images of the target object from multiple perspectives are collected to obtain an original image set, wherein all images in the original image set constitute a complete image of the target object;

[0017] Based on each image in the original image set, pixel points within the target object region are extracted to obtain a preprocessed image, and then a preprocessed image set is obtained.

[0018] As an implementable manner, the preprocessed image set is subjected to sparse reconstruction processing to obtain a sparse point cloud data set, including the following steps:

[0019] Local feature points of each image in the preprocessed image set are extracted to obtain a plurality of local feature point sets;

[0020] Adjacent two images in the preprocessed image set are combined to form an image pair, based on the image pair, local feature points in the local feature point set of the image pair are matched to obtain a matching point pair set;

[0021] The intrinsic matrix of the image acquisition device is point-multiplied with the matching point pairs in the matching point pair set to realize normalization processing, and a normalized matching point pair set is obtained;

[0022] Based on the matching point pairs, the three-dimensional point cloud data, the rotation matrix and the translation vector of the image acquisition device relative to the target object, a three-dimensional projection model is constructed;

[0023] Based on the normalized matching point pair set, combined with the triangulation algorithm or library function, the three-dimensional point cloud data in the three-dimensional projection model is solved, and the obtained three-dimensional point cloud data set is the sparse point cloud data set;

[0024] Wherein, the three-dimensional projection model is represented as follows:

[0025]

[0026] Wherein, Both represent the depth factor, represents two matching points of the th matching point pair in the normalized matching point pair set, represents the three-dimensional point cloud data, represents the intrinsic matrix of the image acquisition device, represents the rotation matrix of the image acquisition device relative to the target object, represents the translation vector of the image acquisition device relative to the target object.

[0027] As an implementable manner, the dense reconstruction processing on the sparse point cloud dataset to obtain the dense point cloud dataset comprises the following steps:

[0028] Project each pixel point in the sparse point cloud dataset to an adjacent view angle to obtain a candidate depth value, and further obtain a projected point depth map;

[0029] Calculate a consistency measure value of each pixel point in the projected point depth map to obtain a consistency measure value set;

[0030] When a consistency measure value of each pixel point in the consistency measure value set reaches a minimum value, the obtained depth value is a depth estimation value of each pixel point, and further an estimated depth map is obtained;

[0031] Fuse the depth value of each pixel point in the estimated depth map to obtain a fused depth map;

[0032] Convert the pixel points in the fused depth map to a three-dimensional space to obtain a dense point cloud dataset;

[0033] The fused depth map is represented as follows:

[0034]

[0035] wherein, represents a consistency measure value of the pixel point in the consistency measure value set, represents a set of projected points of the pixel point in the adjacent view angle, represents a projected point of the pixel point on the view angle , represents a weight function of the pixel point , represents an image intensity of the pixel point in the image corresponding to the view angle , represents an image intensity of the pixel point in the image corresponding to the view angle , represents a fused depth value of the pixel point in the fused depth map, represents a set of all view angles in which the pixel point is visible, represents a confidence weight of the pixel point in the image corresponding to the view angle , represents a depth estimation value of the pixel point in the image corresponding to the view angle .

[0036] As an implementable manner, the scale restoration and posture correction processing on the dense point cloud data set is to obtain a converted dense point cloud data set, and includes the following steps:

[0037] Based on the surface images of the calibration board collected by the image acquisition device from multiple perspectives, a set of original calibration board images is obtained; the pixel coordinates of the feature points in the set of original calibration board images are extracted to obtain a set of feature points; and based on the set of feature points, a rotation matrix and a translation vector of the image acquisition device relative to the calibration board are obtained; wherein all the images of the set of original calibration board images constitute a complete image of the calibration board;

[0038] The model size matrix and the actual size matrix of the calibration board are obtained, and based on the model size matrix and the actual size matrix, a scale factor matrix is obtained;

[0039] Based on the scale factor matrix, the rotation matrix and the translation vector of the image acquisition device relative to the calibration board, a similarity transformation matrix is formed;

[0040] Based on the similarity transformation matrix, the dense point cloud data set is processed to obtain a set of point cloud homogeneous coordinates, and then a converted dense point cloud data set is obtained.

[0041] As an implementable manner, the HSV color space threshold-based filtering processing on the converted dense point cloud data set is to obtain a filtered dense point cloud data set, and includes the following steps:

[0042] The points with HSV values within a preset HSV threshold range in the converted dense point cloud data set are extracted to obtain a target dense point cloud data set;

[0043] The target dense point cloud data set is subjected to clustering processing and the geometric data of each cluster is calculated to obtain a target dense point cloud data geometry set, wherein the geometric data at least includes size, density and geometric shape;

[0044] The points with geometric data within a preset geometric data threshold range in the target dense point cloud data geometry set are extracted to obtain a filtered dense point cloud data set.

[0045] As an implementable manner, the Gaussian optimization and dense reconstruction processing on the sparse point cloud data set is to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, a 3DGS model is obtained by combining a 3DGS reconstruction technology, and then a 3DGS model point cloud data set is obtained, and includes the following steps:

[0046] The sparse point cloud data set is initialized based on Gaussian distribution to obtain a preliminary Gaussian sparse point cloud data set;

[0047] Based on a projection function, the coordinate data of the three-dimensional points in the preliminary Gaussian sparse point cloud data set is predicted to obtain a set of point cloud projection predicted coordinates;

[0048] Obtaining the projection coordinate data of the three-dimensional points in the preliminary Gaussian sparse point cloud data set in the adjacent images to obtain a point cloud projection coordinate set;

[0049] Based on the point cloud projection prediction coordinate set and the point cloud projection coordinate set, a prediction error value set is obtained;

[0050] Based on the minimum prediction error value principle, the optimal coordinate data of the three-dimensional points is obtained, and then an optimized Gaussian sparse point cloud data set is obtained;

[0051] Based on the three-dimensional points in the optimized Gaussian sparse point cloud data set, a plurality of Gaussian distribution three-dimensional points with consistent perspective and consistent depth with the target three-dimensional point are generated to obtain a preliminary Gaussian dense point cloud data of the target three-dimensional point, and then a preliminary Gaussian dense point cloud data set is obtained; the three-dimensional points in the preliminary Gaussian dense point cloud data set are optimized based on the minimum prediction error value to obtain an optimized Gaussian dense point cloud data set;

[0052] The optimized Gaussian dense point cloud data set is post-processed to obtain a Gaussian dense point cloud data set, wherein the post-processing at least includes denoising, interpolation, and hole filling;

[0053] Based on the Gaussian dense point cloud data set, a 3DGS model of the target object is obtained by combining a 3DGS reconstruction technology, and then a 3DGS model point cloud data set is obtained.

[0054] As an implementable manner, the denoising method of the target object 3DGS model further includes removing the three-dimensional points with RGB values exceeding a preset RGB threshold in the denoised 3DGS model to obtain a second denoised 3DGS model, including the following steps:

[0055] Obtaining the spherical harmonic coefficients of the three-dimensional points in the denoised 3DGS model, i.e., the first denoised 3DGS model, to obtain a spherical harmonic coefficient point cloud data set;

[0056] Converting the spherical harmonic coefficients of the three-dimensional points in the spherical harmonic coefficient point cloud data set into RGB values to obtain an RGB point cloud data set;

[0057] Performing clustering processing on the three-dimensional points in the RGB point cloud data set based on the RGB values, if the difference between the RGB value of a three-dimensional point and the RGB values of the three-dimensional points within a preset neighborhood range in the cluster cluster exceeds a preset RGB threshold, then the three-dimensional points with the difference exceeding the preset RGB threshold in the first denoised 3DGS model are removed to obtain a color denoised 3DGS model, i.e., a second denoised 3DGS model.

[0058] A denoising system of a target object 3DGS model includes a preprocessing module, a sparse point cloud module, a dense point cloud module, a dense point cloud filtering module, a 3DGS model module, and a 3DGS model denoising module;

[0059] The preprocessing module acquires original images of the target object from multiple perspectives, obtains an original image set and performs preprocessing to obtain a preprocessed image set, wherein all images in the original image set constitute a complete image of the target object;

[0060] The sparse point cloud module performs sparse reconstruction processing based on the preprocessed image set to obtain a sparse point cloud data set;

[0061] The dense point cloud module performs dense reconstruction processing on the sparse point cloud data set to obtain a dense point cloud data set;

[0062] The dense point cloud filtering module performs scale recovery and posture correction processing on the dense point cloud data set to obtain a rectified dense point cloud data set, and performs filtering processing based on an HSV color space threshold on the rectified dense point cloud data set to obtain a filtered dense point cloud data set;

[0063] The 3DGS model module performs Gaussian optimization and dense reconstruction processing on the sparse point cloud data set to obtain a Gaussian dense point cloud data set, and obtains a 3DGS model based on the Gaussian dense point cloud data set in combination with a 3DGS reconstruction technology, and further obtains a 3DGS model point cloud data set;

[0064] The 3DGS model denoising module calculates distances between corresponding three-dimensional points in the filtered dense point cloud data set and the 3DGS model point cloud data set, removes three-dimensional points in the 3DGS model with distances greater than a preset distance threshold, and obtains a denoised 3DGS model.

[0065] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as follows:

[0066] Acquire original images of the target object from multiple perspectives, obtain an original image set and perform preprocessing, and further obtain a preprocessed image set, wherein all images in the original image set constitute a complete image of the target object;

[0067] Perform sparse reconstruction processing on the preprocessed image set to obtain a sparse point cloud data set;

[0068] Perform dense reconstruction processing on the sparse point cloud data set to obtain a dense point cloud data set;

[0069] Perform scale recovery and posture correction processing on the dense point cloud data set to obtain a rectified dense point cloud data set, and perform filtering processing based on an HSV color space threshold on the rectified dense point cloud data set to obtain a filtered dense point cloud data set;

[0070] The sparse point cloud data set is subjected to Gaussian optimization and dense reconstruction processing to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, a 3DGS model is obtained by combining a 3DGS reconstruction technology, and then a 3DGS model point cloud data set is obtained;

[0071] The distances between corresponding three-dimensional points in the filtered dense point cloud data set and the 3DGS model point cloud data set are calculated, three-dimensional points in the 3DGS model with distances greater than a preset distance threshold are removed, and a denoised 3DGS model is obtained.

[0072] A denoising device for a target object 3DGS model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the method as follows when executing the computer program:

[0073] Original images of the target object from multiple perspectives are obtained to obtain an original image set and perform preprocessing, and then a preprocessed image set is obtained, wherein all images in the original image set constitute a complete image of the target object;

[0074] The preprocessed image set is subjected to sparse reconstruction processing to obtain a sparse point cloud data set;

[0075] The sparse point cloud data set is subjected to dense reconstruction processing to obtain a dense point cloud data set;

[0076] The dense point cloud data set is subjected to scale restoration and pose correction processing to obtain a converted dense point cloud data set; the converted dense point cloud data set is subjected to filtering processing based on an HSV color space threshold to obtain a filtered dense point cloud data set;

[0077] The sparse point cloud data set is subjected to Gaussian optimization and dense reconstruction processing to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, a 3DGS model is obtained by combining a 3DGS reconstruction technology, and then a 3DGS model point cloud data set is obtained;

[0078] The distances between corresponding three-dimensional points in the filtered dense point cloud data set and the 3DGS model point cloud data set are calculated, three-dimensional points in the 3DGS model with distances greater than a preset distance threshold are removed, and a denoised 3DGS model is obtained.

[0079] The present application has the following technical effects: firstly, the hierarchical processing of sparse reconstruction and dense reconstruction is used to construct the geometric skeleton and detail filling, and the scale recovery and posture correction are combined to eliminate the pose deviation, thereby providing a robust foundation for subsequent denoising; secondly, the HSV color space threshold filtering is introduced, the hue (H), saturation (S) and lightness (V) channel separation characteristics are used to accurately remove the reflection noise points and low signal-to-noise ratio areas, and the color-noise coupling problem in the traditional geometric filtering is solved; further, the 3DGS model is generated based on the Gaussian optimization to reserve high-frequency details, and the cross-modal three-dimensional coordinate distance comparison between the filtered dense point cloud and the 3DGS model point cloud is performed to dynamically remove the three-dimensional points deviating from the threshold, correct the geometric deviation caused by the local overfitting of the 3DGS, and realize the collaborative optimization of the global geometric constraint and the local detail enhancement. The present application can not only effectively remove the noise, but also optimize the quality of the three-dimensional model, and meet the demand for high-fidelity three-dimensional reconstruction in various application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0081] Figure 1 is the overall flowchart of the method of the embodiment of the present application;

[0082] Figure 2 is the working state diagram of the image acquisition device collecting the target object image of the embodiment of the present application;

[0083] Figure 3 is the schematic diagram of the original image set of the embodiment of the present application;

[0084] Figure 4 is the three-dimensional dense reconstruction effect diagram of the target object of the embodiment of the present application;

[0085] Figure 5 is the point cloud filtering effect diagram of the embodiment of the present application;

[0086] Figure 6 is the 3DGS reconstruction effect diagram of the embodiment of the present application;

[0087] Figure 7 is the Gaussian filtering effect diagram of the embodiment of the present application;

[0088] Figure 8 is the overall schematic diagram of the system of the embodiment of the present application;

[0089] Reference signs: 1, target object; 2, turntable; 3, camera; 4, camera fixing part; 100, preprocessing module; 200, sparse point cloud module; 300, dense point cloud module; 400, dense point cloud filtering module; 500, 3DGS model module; 600, 3DGS model denoising module. DETAILED DESCRIPTION

[0090] The application will be further described below in conjunction with the embodiments. The following embodiments are explanations of the application and the application is not limited to the following embodiments. The features in the following embodiments can be combined with each other without conflict.

[0091] Embodiment 1

[0092] A denoising method of a target object 3DGS model, as shown in the figure, comprises the following steps: Figure 1

[0093] S100: Obtain original images of the target object from multiple perspectives to obtain an original image set and perform preprocessing to obtain a preprocessed image set, wherein all images in the original image set constitute a complete image of the target object;

[0094] S200: Perform sparse reconstruction processing based on the preprocessed image set to obtain a sparse point cloud data set;

[0095] S300: Perform dense reconstruction processing on the sparse point cloud data set to obtain a dense point cloud data set;

[0096] S400: Perform scale recovery and posture correction processing on the dense point cloud data set to obtain a converted dense point cloud data set; perform filtering processing based on the HSV color space threshold on the converted dense point cloud data set to obtain a filtered dense point cloud data set;

[0097] S500: Perform Gaussian optimization and dense reconstruction processing on the sparse point cloud data set to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, combine 3DGS reconstruction technology to obtain a 3DGS model, and further obtain a 3DGS model point cloud data set;

[0098] S600: Calculate the distance between corresponding three-dimensional points in the filtered dense point cloud data set and the 3DGS model point cloud data set, remove three-dimensional points in the 3DGS model with a distance greater than a preset distance threshold, and obtain a denoised 3DGS model.

[0099] In S100, the original images of the target object from multiple perspectives are obtained to obtain an original image set and perform preprocessing to obtain a preprocessed image set, comprising the following steps:

[0100] ​S110: Collecting surface images of the target object from multiple perspectives by the image collection device to obtain a raw image set, wherein all images in the raw image set constitute a complete image of the target object.

[0101] As shown in Figure 2 Fig. 1 is a schematic diagram of the working state of the image collection device of the present application collecting images of the target object 1, which includes a turntable 2, a camera 3 and a camera fixing component 4. The turntable 2 is a flat turntable, which has a rotation speed control system (not shown in the figure) capable of adjusting the uniform angular velocity, so as to ensure that the rotation speed of the turntable remains consistent during rotation. The camera 3 is fixed on one side of the turntable 2 by the camera fixing component 4, and the camera 3 is preferably a high-frame industrial camera.

[0102] The image collection device collects images of the target object 1, which includes the following steps:

[0103] (1) Fix the target object 1 at the center position of the turntable 2 to ensure that the target object remains vertical during rotation.

[0104] (2) Align the camera 3 with the target object 1 and adjust the focal length and aperture of the camera to ensure that each surface detail of the target object can be clearly and completely captured.

[0105] (3) Start the turntable and rotate it at a constant angular velocity, with a rotation angle of 360 degrees. During rotation, the camera automatically collects images of the target object at certain time intervals to ensure that images of the target object at each angle are collected. After the turntable completes a complete 360-degree rotation, stop the image collection of the camera and place the turntable in a stationary state to obtain a raw image set of the target object.

[0106] As shown in Figure 3 Fig. 2 is a schematic diagram of the effect of the complete perspective image set collected by the camera, which is the raw image set. All images in the raw image set constitute a complete perspective image set of the target object, which covers all directions of the target object. If there are M perspectives that constitute a complete perspective for image collection of the target object, there are M images in the raw image set, and each image is a three-dimensional matrix.

[0107] The original image set collected by the image collection device includes: the view of the target object at each angle during 360-degree rotation, ensuring that all external surfaces (visible parts) of the target object are recorded; by adjusting the camera focal length and aperture, the image can clearly present the details of the texture, color, geometry, label or identification of the surface of the target object; each image corresponds to a fixed angle interval of the rotation of the turntable (determined by the constant angular velocity and the timing shooting of the high-frame camera), for example, if shooting once every 10 degrees, the image set will contain 36 evenly distributed angle images; through the rotation speed control and shooting timestamp of the turntable, the rotation angle of the object corresponding to each image can be indirectly calculated; the images are arranged in chronological order, reflecting the continuous change of the target object during rotation; all images are collected under the same aperture, focal length, and illumination (assuming the light source is fixed), ensuring consistency of brightness, depth of field, and viewing angle. Because the target object is always located at the center of the turntable and remains vertical, the camera is aligned with the center for shooting, which can minimize perspective distortion.

[0108] For specific points in the image set, the collected information includes: the two-dimensional pixel coordinates of the same point in multiple images at different rotation angles, which can infer the three-dimensional spatial position of the point combined with multi-view projection; depth information; the specific rotation angle of the turntable corresponding to each image frame; texture, RGB color, and illumination changes of the same point under different viewing angles.

[0109] S120: performing mask technology processing on the original image set to obtain a mask matrix of the target object, including the following steps:

[0110] The mask technology is applied to the images in the original image set to cover up the background noise or any area that is not conducive to subsequent feature extraction and feature matching in the images. The goal of mask processing is to remove irrelevant parts in the image to ensure that only the effective image information of the target object is retained. The mask can be in the form of a rectangle, a circle, or other shapes designed according to the outline of the object. The mask area is represented by a mask matrix, which is a two-dimensional matrix. The mask matrix has the same size as the matrix corresponding to the images in the original image set. Each element value in the mask matrix is usually 0 or 1, indicating whether the corresponding image area should be processed or retained. The mask matrix is represented as follows:

[0111]

[0112] wherein, represents the value of the mask matrix at the coordinate / pixel point , the mask matrix determines whether the pixel point whether in the target object region, 1 indicates that the pixel point is in the target object region and is retained, 0 indicates that the pixel point is shielded (not in the target object region) and is abandoned. Each image corresponds to a mask, and the mask is a two-dimensional matrix consistent with the height and width of each image. The mask matrix is a three-dimensional matrix composed of M two-dimensional matrices consistent with the original image set.

[0113] S130: obtaining a pre-processed image set based on the original image set and the mask matrix wherein, represents the pixel value of the image in the pre-processed image set at coordinates , represents the pixel value of the image in the original image set at coordinates , and the target object region is extracted by pixel-by-pixel multiplication.

[0114] In S200, sparse reconstruction processing is performed on the pre-processed image set to obtain a sparse point cloud data set

[0115] Sparse reconstruction refers to the process of recovering the sparse three-dimensional point cloud structure of a scene and the camera pose from multiple two-dimensional images. One or more of the following sparse reconstruction algorithms and frameworks can be selected to implement the sparse reconstruction processing on the pre-processed image set described in the present application: traditional SFM (Structure from Motion) algorithm or PnP (Perspective-n-Point) algorithm that extracts feature points in the image for matching and then calculates the three-dimensional structure by multi-view geometry principle, incremental SFM algorithm or global SFM algorithm based on incremental / global optimization, improved SFM algorithm based on deep learning. The choice of algorithm for sparse reconstruction processing is determined by the demand scenario and data size, etc.

[0116] In this embodiment, after the original image set is pre-processed to obtain the pre-processed image set, the SFM algorithm is used to specifically explain the process of sparse reconstruction, but it is not limited to the algorithm process described in this embodiment. Sparse reconstruction processing is performed on the pre-processed image set to obtain a sparse point cloud data set, including the following steps:

[0117] S210: extracting local feature points of each image in the pre-processed image set to obtain a plurality of local feature point sets. Common methods include SIFT, SURF, ORB, etc. Let the feature points in image 1 and image 2 be and wherein, and represent the index of the feature point.

[0118] S220: two adjacent images in the pre-processed image set are combined into an image pair, local feature points in a local feature point set of the image pair are matched based on the image pair, and a matching point pair set is obtained.

[0119] Feature matching is to find matching feature point pairs between image pairs. Let the matching point pair be , that is, it is assumed that the first feature points are visible in both images and have been correctly matched.

[0120] S230: based on the intrinsic matrix of the image acquisition device, the matching point pairs in the matching point pair set are normalized to obtain a normalized matching point pair set. The normalized matching point pair set is represented as and , wherein represents two matching points of the first matching point pair in the normalized matching point pair set, that is, the normalized point, and the normalized point has removed the influence of the intrinsic parameter, represents two matching points of the first matching point pair in the matching point pair set, represents the intrinsic matrix of the image acquisition device, and the intrinsic matrix is a property (such as focal length, principal point) of the image acquisition device itself, which is irrelevant to the position of the image acquisition device.

[0121] S240: based on the matching point pair, the three-dimensional point cloud data, the rotation matrix and the translation vector of the image acquisition device relative to the target object, a three-dimensional projection model is constructed; the three-dimensional projection model is represented as follows:

[0122]

[0123] , wherein both represent a depth factor (i.e., the distance from the point to the image acquisition device), represents the rotation matrix of the image acquisition device, represents the translation vector of the image acquisition device, represents the three-dimensional point cloud data. The rotation matrix and the translation vector are collectively referred to as the extrinsic matrix of the image acquisition device relative to the target object.

[0124] S250: based on the normalized matching point pair set, combined with a triangulation algorithm or a library function, the three-dimensional point cloud data in the three-dimensional projection model is iteratively solved based on the principle of minimizing the projection error, and the obtained three-dimensional point cloud data set is the sparse point cloud data set.

[0125] The point cloud data in the final sparse point cloud dataset corresponds to the matching feature points of the images in the preprocessed image set. The above processing process only provides a reference, and can also be adjusted and optimized according to specific conditions.

[0126] In S300, the sparse point cloud dataset is subjected to dense reconstruction processing to obtain a dense point cloud dataset

[0127] The dense reconstruction recovers the dense point cloud of the scene from the sparse point cloud. The mainstream dense reconstruction algorithm includes the MVS algorithm for generating a dense point cloud or a depth map through pixel matching and depth estimation of multiple images, the fusion of a single or multiple depth maps into a unified dense model (including KinectFusion, ElasticFusion, BundleFusion, and Open3D, etc.), and the completion or conversion of the sparse point cloud into a dense surface (including Poisson reconstruction, Screened Poisson Reconstruction, and BPA algorithm, etc.). In actual applications, a corresponding dense reconstruction algorithm can be selected as needed.

[0128] In this embodiment, a dense reconstruction method based on the MVS algorithm is provided, and the process of dense reconstruction is specifically described, but is not limited to the algorithm process described in this embodiment. The sparse point cloud dataset is subjected to dense reconstruction processing to obtain a dense point cloud dataset, including the following steps:

[0129] S310: Project each pixel point in the sparse point cloud dataset to an adjacent view angle to obtain a candidate depth value, and further obtain a projection point depth map, that is, a matrix composed of candidate depth values. Specifically, search for the corresponding point of each pixel point in view angle i in adjacent view angle j through epipolar geometry to generate a candidate depth value; and combine the candidate depth values of all pixel points according to the corresponding coordinate data into a matrix form, that is, a projection point depth map.

[0130] S320: Calculate the consistency measure value of each pixel point in the projection point depth map to obtain a consistency measure value set. The consistency measure value represents the consistency of the pixel point under a given depth (from the projection point depth map) assumption. The consistency measure value set is represented as: wherein, represents the consistency measure value of the pixel point in the consistency measure value set, represents the projection point set of the pixel point in the adjacent view angle, represents the projection point of the pixel point in view angle , represents the weight function of the pixel point , represents the view angle the image intensity of the pixel point in the corresponding image, the viewing angle, the image intensity of the pixel point in the corresponding image,

[0131] S330: when the consistency measure value of each pixel point in the consistency measure value set reaches the minimum value, the obtained depth value is the depth estimation value of each pixel point, and then an estimated depth map is obtained. Specifically, when the consistency measure value of the pixel point in the consistency measure value set reaches the minimum value, the obtained depth value is the depth estimation value of the pixel point , and the depth estimation values of all pixel points are obtained by the same method, and the depth estimation values of all pixel points are combined into a matrix form according to the corresponding coordinate data to obtain the estimated depth map.

[0132] S340: fusing the depth values of each pixel point in the estimated depth map to obtain a fused depth map. The fused depth map is a dense point cloud obtained by fusing all reference depth maps, and a weighted average strategy can be used to process multiple observations. The fused depth map is represented as follows:

[0133]

[0134] wherein, denotes the fused depth value of the pixel point in the fused depth map, denotes the set of all viewing angles that can see the pixel point , denotes the confidence weight of the pixel point in the corresponding image of the viewing angle , and denotes the depth estimation value of the pixel point in the corresponding image of the viewing angle .

[0135] S350: the pixel points in the fused depth map can be converted to a three-dimensional space by technologies such as direct projection (Open3D / PCL), COLMAP / MVS process, KinectFusion / ORB-SLAM3 (with a dense module), depth completion network (CSPN / NLSPN), etc., to obtain a dense point cloud dataset.

[0136] The above steps perform dense reconstruction processing on the sparse point cloud dataset, use consistency matching and depth estimation in multi-view images to fill the missing areas in the sparse point cloud dataset, and finally obtain point cloud data in the dense point cloud dataset, which contains three-dimensional coordinates, color, etc. of each pixel point, such as Figure 4 ​​​​​The effect diagram after the three-dimensional dense reconstruction of the target object is shown.

[0137] In S400, the scale recovery and pose correction processing is performed on the dense point cloud data set to obtain a rectified dense point cloud data set; the filtering processing based on the HSV color space threshold is performed on the rectified dense point cloud data set to obtain a filtered dense point cloud data set

[0138] S410: performing scale recovery and pose correction processing on the dense point cloud data set to obtain a rectified dense point cloud data set, including the following steps:

[0139] (1) acquiring surface images of the calibration board at multiple viewing angles based on the image acquisition device to obtain a calibration board original image matrix set; extracting pixel coordinates of feature points in the calibration board original image matrix set to obtain a feature point matrix set; and obtaining a rotation matrix and a translation vector of the image acquisition device relative to the calibration board based on the feature point matrix set, wherein all images in the calibration board original image matrix set constitute a complete image of the calibration board.

[0140] The calibration board is usually a planar board with a known geometric shape (for example, a checkerboard). The image acquisition can be performed at multiple angles and multiple viewing distances. During acquisition, it is ensured that each corner and the middle area of the calibration board captured by the image acquisition device is clearly visible, so as to facilitate subsequent calibration calculation. The rotation matrix and the translation vector of the image acquisition device relative to the calibration board can be obtained by conventional means, and will not be described herein.

[0141] (2) obtaining a model size matrix and an actual size matrix of the calibration board, and obtaining a scale factor matrix based on the model size matrix and the actual size matrix; the scale factor matrix is represented as wherein, the scale factor matrix is a diagonal scale matrix, and the diagonal scale matrix is a matrix of the actual size matrix of the calibration board in the physical space, and the model size of the calibration board in the world coordinate system.

[0142] (3) forming a similarity transformation matrix based on the scale factor matrix, the rotation matrix and the translation vector of the image acquisition device relative to the calibration board; the similarity transformation matrix is represented as wherein, the rotation matrix of the image acquisition device relative to the calibration board is a matrix of and is also a matrix of . the translation vector of the image acquisition device relative to the calibration board is a matrix of . The in the last row is used for normalization of the homogeneous coordinates. ​denotes a similarity transformation matrix, is a matrix.

[0143] (4) Based on the similarity transformation matrix, the dense point cloud data set is transformed to obtain a point cloud homogeneous coordinate set, and then a rectified dense point cloud data set is obtained; the point cloud homogeneous coordinate set is represented as: , wherein, represents the i-th homogeneous coordinate in the point cloud homogeneous coordinate set, represents the i-th three-dimensional coordinate in the dense point cloud data set, represents the i-th three-dimensional coordinate in the rectified dense point cloud data set.

[0144] The similarity transformation matrix is used to convert the target object model from the calibration coordinate system to the actual physical coordinate system, and make its size and posture consistent with that in the actual physical space, to realize scale recovery and posture correction.

[0145] S420: The rectified dense point cloud data set is filtered based on the HSV color space threshold to obtain a filtered dense point cloud data set, including the following steps:

[0146] (1) Extracting points in the rectified dense point cloud data set whose HSV values are within a preset HSV threshold range to obtain a target dense point cloud data set. Convert the RGB values in the rectified dense point cloud data set to HSV values, use a preset HSV threshold range for segmentation as a color segmentation algorithm, compare the HSV values of each pixel point in the rectified dense point cloud data set with the preset HSV threshold range, if the HSV value of the pixel point is within the preset HSV threshold range, the pixel point is regarded as a target point; otherwise, it is regarded as an outlier. Therefore, the target dense point cloud data set only includes target points.

[0147] (2) Using a point cloud clustering algorithm (such as K-means), the target dense point cloud data set is clustered to obtain a plurality of clusters ; and calculating the geometric data of each cluster, the geometric data at least including size, density and geometric shape, to obtain a target dense point cloud data geometry set.

[0148] (3) Extracting points in the target dense point cloud data geometry set whose geometric data (size, density and geometric shape, etc.) are within a preset geometric data threshold range, removing clusters that do not meet the conditions, to obtain a filtered dense point cloud data set to realize point cloud filtering. As shown in Figure 5 , it is an effect diagram after point cloud filtering of the embodiment

[0149] ​​​​The various thresholds or threshold ranges preset in the present application can be directly set based on historical experience or field expert suggestions, can be set according to quantiles of data distribution, or can adopt dynamic thresholds, and the present application is not limited.

[0150] In S500, Gaussian optimization and dense reconstruction processing are performed on the sparse point cloud data set to obtain a Gaussian dense point cloud data set; based on the Gaussian dense point cloud data set, a 3DGS model is obtained by combining a 3DGS reconstruction technology, and then a 3DGS model point cloud data set is obtained, including the following steps:

[0151] S510: The sparse point cloud data set is initialized based on Gaussian distribution to obtain a preliminary Gaussian sparse point cloud data set. The pixel points of the point cloud data after Gaussian distribution initialization processing are subject to Gaussian distribution, that is, wherein, is the spatial coordinates of the pixel point , the uncertainty of the pixel point , and the uncertainty can be estimated by the perspective distribution of the image and the matching error.

[0152] After the sparse point cloud data set is initialized based on Gaussian distribution, an optimization algorithm is used to adjust the Gaussian distribution model of each pixel point to minimize the prediction error value. The optimization algorithm that can be used includes: gradient descent-based algorithms (such as basic gradient descent algorithm, stochastic gradient descent algorithm SGD, Adam / AdaGrad / RMSProp adaptive learning gradient descent algorithm), second-order optimization algorithms (such as nonlinear optimization algorithm Bundle Adjustment, Newton method, Gauss-Newton method, combination of Gauss-Newton and gradient descent LM, etc.), robust optimization algorithms, global optimization algorithms (such as particle swarm optimization PSO, genetic algorithm GA, simulated annealing algorithm Simulated Annealing, etc.), distributed / incremental algorithms, and methods combined with deep learning, etc. In actual application, the above algorithms should be flexibly selected or combined according to the specific scene (data size, noise level, real-time requirement, etc.). In the present application, the nonlinear optimization algorithm Bundle Adjustment is taken as an example to illustrate the optimization process, and those skilled in the art should know that the optimization algorithm that can be used in the present application is not limited to the nonlinear optimization algorithm BA.

[0153] S520: Based on the projection function, the coordinate data of the three-dimensional points in the preliminary Gaussian sparse point cloud data set is predicted to obtain a point cloud projection prediction coordinate set. The prediction coordinate data of the three-dimensional point is wherein, ​This represents the projection function. The type of projection function is determined by the type of image acquisition device, such as pinhole model, fisheye model, or orthographic projection. This represents the intrinsic parameter matrix of the image acquisition device. Indicates the image acquisition device number The rotation matrix of each viewpoint relative to the target object. Indicates the image acquisition device number The translation vector of a viewpoint relative to the target object.

[0154] S530: Obtain the projection coordinates of 3D points in the preliminary Gaussian sparse point cloud dataset onto neighboring images to obtain a point cloud projection coordinate set. Each 3D point in the preliminary Gaussian sparse point cloud dataset is visible in at least one original image. For each 3D point, based on camera pose or co-viewing relationships, select several viewpoints adjacent to the original observation image of that point (e.g., images with small angles or close baselines). Then, using the camera extrinsic parameters (rotation and translation matrices) of the neighboring images, transform the 3D point to the coordinate system of each neighboring camera, and then calculate its 2D projection coordinates on the image using camera intrinsic parameters (focal length, principal point, etc.). Further, check whether the projected points are within the image boundaries and eliminate erroneous projections caused by occlusion or invalid depth (e.g., points behind the camera). Finally, store the 2D projection coordinates of each 3D point in its visible neighboring images to form a structured point cloud projection coordinate set.

[0155] S540: Based on the point cloud projection prediction coordinate set and the point cloud projection coordinate set, the reprojection error value set is obtained, which is the prediction error value set. The prediction error value set is represented as follows: ,in, This indicates the first [item] in the initial Gaussian sparse point cloud dataset. Coordinate data of three-dimensional points, Represents the set of prediction error values. Three-dimensional points The prediction error value, This represents the total number of viewpoints when the image acquisition device captures images of the target object. Representing a three-dimensional point In the The first perspective (that is, the first) The projection coordinates in the image corresponding to each viewpoint, Represents the projection function. Represents the first point in the point cloud projection prediction coordinate set. Predicted coordinate data of three-dimensional points The norm is represented. By minimizing the prediction error, the position and uncertainty of each 3D point are updated to obtain the optimal coordinate data for each 3D point, thereby optimizing the Gaussian distribution model.

[0156] S550: When the prediction error value of the target three-dimensional point is the smallest in the prediction error value set, the optimal coordinate data of the three-dimensional point is obtained, and then the optimized Gaussian sparse point cloud data set is obtained. Specifically, based on the principle of minimizing the prediction error value, the three-dimensional point with the smallest prediction error value is obtained, and the coordinate data of the three-dimensional point is the optimal coordinate data. The optimal coordinate data of other three-dimensional points is obtained in the same way. The optimal coordinate data of all three-dimensional points replaces the coordinate data of the corresponding three-dimensional point in the preliminary Gaussian sparse point cloud data set, forming the optimized Gaussian sparse point cloud data set.

[0157] S560: Based on the optimized sparse point cloud, more Gaussian point clouds can be generated by detailed analysis of the surrounding area, which is called point cloud densification. For each optimized three-dimensional point , the area near the optimized spatial position is used to generate more points based on the principles of perspective-based photometric consistency and depth consistency. These newly generated points are assigned similar Gaussian distributions to the original points, and the density of the point cloud is updated. Each new point is also represented by a Gaussian distribution model, and is further optimized by minimizing the new prediction error.

[0158] Point cloud densification includes: based on the three-dimensional points in the optimized Gaussian sparse point cloud data set, a number of Gaussian distribution three-dimensional points consistent with the perspective and depth of the target three-dimensional point are generated, and a preliminary Gaussian dense point cloud data set is obtained; the three-dimensional points in the preliminary Gaussian dense point cloud data set are optimized based on the minimization of the prediction error value, and an optimized Gaussian dense point cloud data set is obtained. Specifically, point cloud densification can be achieved by the following steps:

[0159] (1) Perspective modeling. An observation sphere is constructed with the optimized three-dimensional point as the center, and a number of virtual perspectives are uniformly sampled on the sphere (such as Fibonacci sphere sampling). Each virtual perspective corresponds to a virtual camera, and its optical axis points to the original three-dimensional point, maintaining similar camera parameters to the original observation

[0160] (2) Photometric consistency modeling. For each virtual perspective: the predicted color under the perspective is obtained by differentiable rendering (using the original image for bilinear interpolation); a color Gaussian distribution is established, with the mean = predicted color and the variance determined by image noise and interpolation error; the color distribution of the newly generated point is constrained to be consistent with the predicted distribution by KL divergence.

[0161] (3) Depth consistency modeling. The theoretical depth value of the original three-dimensional point under the virtual perspective is calculated; a depth Gaussian distribution is constructed, with the mean = theoretical depth value and the variance determined by the point cloud density and the camera baseline length. The depth likelihood constraint is imposed on the generated point to make it distributed in a reasonable depth range.

[0162] (4) Three-dimensional Gaussian point generation. In the initialization, the original point is taken as the center, and the covariance matrix is determined by the perspective distribution and the scene scale. The differentiable three-dimensional Gaussian splatting method can be used to achieve this.

[0163] (5) Optimization. The luminosity loss requires the rendered color to match the predicted distribution, the depth loss requires the projected depth to conform to the theoretical distribution, and the geometric regularization term is used to prevent the Gaussian points from being excessively dispersed. Optimization can be performed through gradient descent.

[0164] (6) After iterative optimization, remove Gaussian points that do not meet the consistency constraints (such as luminosity error / depth error exceeding the threshold), and adjust the density of the remaining Gaussian points to ensure reasonable coverage under perspective changes.

[0165] S570: After the completion of the point cloud densification, post-processing is performed on the optimized Gaussian dense point cloud dataset to obtain a Gaussian dense point cloud dataset, which at least includes denoising, point interpolation, and hole filling. The quality of the generated point cloud is further optimized through the post-processing step.

[0166] S580: Based on the Gaussian dense point cloud dataset, combined with 3DGS reconstruction technology (including original 3DGS framework, dynamic scene expansion (4D Gaussian splatting, motion decomposition), SLAM and real-time reconstruction (such as SplaTAM), large scene and generalization (block optimization, pre-trained model), and multi-modal fusion (semantic segmentation, NeRF combination), etc.), a 3DGS model of the target object is obtained, and then a 3DGS model point cloud dataset is obtained. For example, using GPU-accelerated volume rendering technology to convert the dense point cloud into a high-quality three-dimensional scene model, the process of volume rendering generates the final rendering image by calculating the transparency, color, and lighting of each point, as shown in Figure 6 , which is the effect diagram after 3DGS reconstruction of the present embodiment.

[0167] In S600, the distance between corresponding three-dimensional points in the filtered dense point cloud dataset and the 3DGS model point cloud dataset is calculated, and three-dimensional points in the 3DGS model with a distance greater than a preset distance threshold are removed to obtain a denoised 3DGS model, including the following steps:

[0168] S610: Each point cloud in the 3DGS model is matched with the filtered dense target point cloud point by point, and by comparing the three-dimensional coordinates of each point, points with large differences from the Gaussian distribution in the model, i.e., noise points, are filtered out. The removal of noise points is based on: in space, if the three-dimensional coordinates of a point and the distribution center of the corresponding point's Gaussian model If the distance between the filtered dense point cloud dataset and the 3DGS model point cloud dataset exceeds a certain threshold, the point is considered to be a noise point and is removed. The threshold can be set according to the actual situation. Specifically, the distance between the three-dimensional point coordinates of the corresponding point clouds in the filtered dense point cloud dataset and the 3DGS model point cloud dataset is calculated. If the distance is greater than a predetermined distance threshold, the three-dimensional point corresponding to the distance is removed from the 3DGS model to obtain a distance denoised 3DGS model point cloud dataset, and then a first denoised 3DGS model is obtained.

[0169] After obtaining the first denoised 3DGS model based on distance-based denoising, color denoising based on spherical harmonic coefficients can be further performed on the first 3DGS model. Spherical harmonic coefficients are mathematical tools for describing the shape of a three-dimensional surface. By converting them into RGB color values, corresponding color information can be generated for each point. Spherical harmonic coefficients can be obtained through spherical harmonic transformation, and then a clustering algorithm (such as K-means) can be used to group points with similar colors together. If the RGB color of a point differs greatly from the colors of its neighboring points, the point belongs to the noise category and is removed. Specifically, the spherical harmonic coefficients of the three-dimensional points in the distance denoised 3DGS model (the first denoised 3DGS model) are obtained to obtain a spherical harmonic coefficient point cloud dataset. In 3DGS, spherical harmonic coefficients are used to represent the change in color or appearance of a three-dimensional point at different viewing angles. Each three-dimensional point may be associated with a set of spherical harmonic coefficients (usually 2nd or 3rd order), which are used to dynamically calculate the color during rendering. The attributes of each three-dimensional point in the first denoised 3DGS model include: position (XYZ), color (RGB), transparency, scale (Scale), rotation (Rotation), and spherical harmonic coefficients (SH coefficients). The spherical harmonic coefficients of the 3DGS model can be directly extracted.

[0170] S620: Obtain the spherical harmonic coefficients of the three-dimensional points in the distance denoised 3DGS model (the first denoised 3DGS model) to obtain a spherical harmonic coefficient point cloud dataset. In 3DGS, spherical harmonic coefficients are used to represent the change in color or appearance of a three-dimensional point at different viewing angles. Each three-dimensional point may be associated with a set of spherical harmonic coefficients (usually 2nd or 3rd order), which are used to dynamically calculate the color during rendering. The attributes of each three-dimensional point in the first denoised 3DGS model include: position (XYZ), color (RGB), transparency, scale (Scale), rotation (Rotation), and spherical harmonic coefficients (SH coefficients). The spherical harmonic coefficients of the 3DGS model can be directly extracted.

[0171] S630: Convert the spherical harmonic coefficients of the three-dimensional points in the spherical harmonic coefficient point cloud dataset to RGB values to obtain an RGB point cloud dataset. Each three-dimensional point typically stores a set of spherical harmonic coefficients, corresponding to the three RGB channels (for example, 3rd order spherical harmonic has 16 coefficients / channels, a total of 48 parameters). The normalized unit direction vector from the camera to the three-dimensional point is the observation direction vector of the three-dimensional point, and the observation direction vector of the three-dimensional point in the Cartesian coordinate system is converted to the pitch angle (polar angle) and azimuth angle in the spherical coordinate system. Based on the pitch angle and the azimuth angle, the pre-defined spherical harmonic basis function (which can be quickly obtained through a formula or a pre-computed table, such as 16 for 3rd order) is used to calculate the spherical harmonic basis function value of the corresponding order. For each color channel (R, G, B), multiply the spherical harmonic coefficients of the channel with the basis function value and sum (weighted sum) to obtain the RGB color value of the channel. Finally, the RGB color value obtained by the weighted sum needs to be truncated to the range.

[0172] S640: RGB value-based clustering processing is performed on the three-dimensional points in the RGB point cloud dataset, if the difference between the RGB value of a three-dimensional point and the RGB value of a three-dimensional point within a preset neighborhood range in the cluster cluster exceeds a preset RGB threshold, the three-dimensional point whose difference exceeds the preset RGB threshold in the first denoising 3DGS model is removed, and a color denoising 3DGS model, i.e., a second denoising 3DGS model, is obtained, as shown in Figure 7 .

[0173] The embodiment proposes to map the spherical harmonic function of the Gaussian sphere to the RGB space, decouple the noise and the real signal by using the color domain filtering technology, combine the pre- denoising three-dimensional dense model, realize geometric consistency constraint by comparing the cross-modal coordinate distance, and thus balance the noise suppression, detail preservation and cross- scene robustness in complex scenes, and significantly improve the reconstruction quality and applicability of the three-dimensional model.

[0174] Embodiment 2:

[0175] A denoising system of a target object 3DGS model, as shown in Figure 8 , includes a preprocessing module 100, a sparse point cloud module 200, a dense point cloud module 300, a dense point cloud filtering module 400, a 3DGS model module 500, and a 3DGS model denoising module 600.

[0176] The preprocessing module 100 acquires original images of a target object from multiple perspectives, obtains an original image set and performs preprocessing to obtain a preprocessed image set.

[0177] The sparse point cloud module 200 performs sparse reconstruction processing based on the preprocessed image set to obtain a sparse point cloud dataset.

[0178] The dense point cloud module 300 performs dense reconstruction processing on the sparse point cloud dataset to obtain a dense point cloud dataset.

[0179] The dense point cloud filtering module 400 performs scale recovery and posture correction processing on the dense point cloud dataset to obtain a converted dense point cloud dataset, and performs filtering processing based on the HSV color space threshold on the converted dense point cloud dataset to obtain a filtered dense point cloud dataset.

[0180] The 3DGS model module 500 performs Gaussian optimization and dense reconstruction processing on the sparse point cloud dataset to obtain a Gaussian dense point cloud dataset, and obtains a 3DGS model based on the Gaussian dense point cloud dataset in combination with 3DGS reconstruction technology, and further obtains a 3DGS model point cloud dataset.

[0181] The 3DGS model denoising module 600 compares the filtered dense point cloud data set and the 3DGS model point cloud data set, removes points in the 3DGS model with a distance greater than a preset threshold, and obtains a denoised 3DGS model.

[0182] Various changes and modifications can be made to the application without departing from the spirit and scope of the application. All equivalent technical solutions are within the scope of the application.

[0183] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between embodiments can be referred to each other.

[0184] Those skilled in the art should understand that the embodiments of the application can be provided as a method, device, or computer program product. Therefore, the application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can be in the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk storage, CD-ROM, optical storage, etc.).

[0185] The application is described with reference to flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the computer or other programmable data processing terminal device produce a device that implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks.

[0186] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction apparatus, which implements the functions specified in the flowchart and / or block diagram. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that performs the functions specified in one or more blocks.

[0187] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operational steps are performed on the computer or other programmable terminal device to generate a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the function specified in the flowchart Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0188] It should be noted that

[0189] The phrase "one embodiment" or "an embodiment" as used throughout this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Therefore, the appearances of the phrase "one embodiment" or "an embodiment" throughout the specification are not necessarily all referring to the same embodiment.

[0190] Furthermore, it is noted that the specific embodiments described in this specification are merely illustrative and that changes can be made in form, detail, and / or function without departing from the spirit of the application. As such, other implementations of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. Moreover, various modifications and changes can be made thereto by those skilled in the art which further fall within the scope and spirit of the application as defined by the appended claims.

Claims

1. A method for denoising a 3DGS model of a target object, characterized in that, Includes the following steps: The original images of the target object from multiple perspectives are obtained, the original image set is preprocessed, and then the preprocessed image set is obtained. The complete image of the target object is composed of all the images in the original image set. Sparse reconstruction processing is performed on the preprocessed image set to obtain a sparse point cloud dataset; Dense point cloud datasets are obtained by performing dense reconstruction on sparse point cloud datasets. Scale restoration and pose correction are performed on the dense point cloud dataset to obtain a normalized dense point cloud dataset; the normalized dense point cloud dataset is then filtered based on HSV color space thresholds to obtain a filtered dense point cloud dataset. Gaussian optimization and dense reconstruction are performed on the sparse point cloud dataset to obtain a Gaussian dense point cloud dataset. Specifically, the sparse point cloud dataset is initialized based on a Gaussian distribution and optimized based on the principle of minimizing prediction error to obtain an optimized Gaussian sparse point cloud dataset; the optimized Gaussian sparse point cloud dataset is then densely reconstructed and optimized based on the principle of minimizing prediction error to obtain a Gaussian dense point cloud dataset. Based on the Gaussian dense point cloud dataset, combined with 3DGS reconstruction technology, a 3DGS model is obtained, and then a 3DGS model point cloud dataset is obtained. Calculate the distance between corresponding 3D points in the filtered dense point cloud dataset and the 3DGS model point cloud dataset, remove 3D points in the 3DGS model whose distance is greater than a preset distance threshold, and obtain a denoised 3DGS model.

2. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of acquiring original images of the target object from multiple perspectives, obtaining a set of original images, and preprocessing them to obtain a preprocessed image set includes the following steps: The surface images of the target object are acquired from multiple perspectives using an image acquisition device to obtain an original image set. All images in the original image set constitute a complete image of the target object. Based on each image in the original image set, the pixels within the target object region are extracted to obtain a preprocessed image, which in turn yields a preprocessed image set.

3. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of performing sparse reconstruction on the preprocessed image set to obtain a sparse point cloud dataset includes the following steps: Local feature points are extracted from each image in the preprocessed image set to obtain multiple local feature point sets; The preprocessed image set is divided into two adjacent images to form an image pair. Based on the image pair, the local feature points in the local feature point set of the image pair are matched to obtain a set of matching point pairs. Normalization is achieved by multiplying the intrinsic parameter matrix of the image acquisition device with the matching point pairs in the matching point pair set, resulting in a normalized matching point pair set. A 3D projection model is constructed based on matching point pairs, 3D point cloud data, and the rotation matrix and translation vector of the image acquisition device relative to the target object. Based on the normalized matching point pair set, combined with the triangulation algorithm or library function, the 3D point cloud data in the 3D projection model is solved, and the resulting 3D point cloud dataset is the sparse point cloud dataset. The three-dimensional projection model is represented as follows: in, Both represent depth factors. Represents the set of normalized matching point pairs. Two matching points in a pair of matching points Represents 3D point cloud data. This represents the intrinsic parameter matrix of the image acquisition device. This represents the rotation matrix of the image acquisition device relative to the target object. This represents the translation vector of the image acquisition device relative to the target object.

4. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of performing dense reconstruction on the sparse point cloud dataset to obtain a dense point cloud dataset includes the following steps: Each pixel in the sparse point cloud dataset is projected onto the adjacent viewpoints to obtain candidate depth values, and then the projection point depth map is obtained. Calculate the consistency metric value for each pixel in the depth map of the projected points to obtain a set of consistency metrics; When the consistency metric value of each pixel in the consistency metric set reaches its minimum value, the resulting depth value is the depth estimate of each pixel, thus obtaining the estimated depth map; The depth values ​​of each pixel in the estimated depth map are fused to obtain the fused depth map; The pixels in the fused depth map are transformed into three-dimensional space to obtain a dense point cloud dataset; The fusion depth map is represented as follows: in, Represents pixels in the fused depth map The fusion depth value, Indicates visible pixels The collection of all perspectives Indicates perspective Corresponding pixels in the image Confidence weights Indicates perspective Corresponding pixels in the image The depth estimate.

5. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of performing scale restoration and pose correction on the dense point cloud dataset to obtain a normalized dense point cloud dataset includes the following steps: The calibration board surface images are acquired from multiple perspectives using an image acquisition device to obtain the original image set of the calibration board; the pixel coordinates of feature points in the original image set of the calibration board are extracted to obtain the feature point set; based on the feature point set, the rotation matrix and translation vector of the image acquisition device relative to the calibration board are obtained; wherein, all images in the original image set of the calibration board constitute the complete image of the calibration board. Obtain the model size matrix and actual size matrix of the calibration board, and based on the model size matrix and actual size matrix, obtain the scale factor matrix; A similarity transformation matrix is ​​formed based on the scale factor matrix, the rotation matrix of the image acquisition device relative to the calibration plate, and the translation vector. Based on the similarity transformation matrix, the dense point cloud dataset is transformed to obtain the homogeneous coordinate set of the point cloud, and then the normalized dense point cloud dataset is obtained.

6. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of filtering the dense point cloud dataset based on the HSV color space threshold to obtain the filtered dense point cloud dataset includes the following steps: Extract points in the dense point cloud dataset whose HSV values ​​are within the preset HSV threshold range to obtain the target dense point cloud dataset; Clustering is performed on the target dense point cloud dataset and the geometric data of each cluster is calculated to obtain the target dense point cloud data geometric set, wherein the geometric data includes at least size, density and geometric shape; Extract points from the geometric set of the target dense point cloud data that fall within the preset geometric data threshold range to obtain a filtered dense point cloud dataset.

7. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, The process of performing Gaussian optimization and dense reconstruction on the sparse point cloud dataset to obtain a Gaussian dense point cloud dataset; based on the Gaussian dense point cloud dataset, combined with 3DGS reconstruction technology, a 3DGS model is obtained, and then a 3DGS model point cloud dataset is obtained, including the following steps: The sparse point cloud dataset is initialized based on a Gaussian distribution to obtain a preliminary Gaussian sparse point cloud dataset. Based on the projection function, the coordinate data of 3D points in the preliminary Gaussian sparse point cloud dataset are predicted to obtain the point cloud projection prediction coordinate set. Obtain the projection coordinate data of 3D points in the preliminary Gaussian sparse point cloud dataset in adjacent images to obtain the point cloud projection coordinate set; Based on the point cloud projection prediction coordinate set and the point cloud projection coordinate set, the prediction error value set is obtained; Based on the principle of minimizing prediction error, the optimal coordinate data of three-dimensional points are obtained, and then an optimized Gaussian sparse point cloud dataset is obtained. Based on the 3D points in the optimized Gaussian sparse point cloud dataset, several Gaussian distributed 3D points with the same light intensity and depth as the target 3D point are generated to obtain the preliminary Gaussian dense point cloud data of the target 3D point, and then the preliminary Gaussian dense point cloud dataset is obtained; the 3D points in the preliminary Gaussian dense point cloud dataset are optimized based on minimizing the prediction error value to obtain the optimized Gaussian dense point cloud dataset. Post-processing is performed on the optimized Gaussian dense point cloud dataset to obtain a Gaussian dense point cloud dataset, wherein the post-processing includes at least denoising, interpolation, and hole filling; Based on the Gaussian dense point cloud dataset, combined with 3DGS reconstruction technology, a 3DGS model of the target object is obtained, and then a 3DGS model point cloud dataset is obtained.

8. The denoising method for a 3DGS model of a target object according to claim 1, characterized in that, It also includes removing 3D points in the denoised 3DGS model whose RGB values ​​exceed a preset RGB threshold to obtain a second denoised 3DGS model, including the following steps: The spherical harmonic coefficients of the three-dimensional points in the first denoised 3DGS model are obtained to obtain the spherical harmonic coefficient point cloud dataset. Convert the spherical harmonic coefficients of the 3D points in the spherical harmonic coefficient point cloud dataset to RGB values ​​to obtain the RGB point cloud dataset; Clustering is performed on the 3D points in the RGB point cloud dataset based on RGB values. If the difference between the RGB value of a 3D point and the RGB value of a 3D point within a preset neighborhood of its cluster exceeds a preset RGB threshold, then the 3D points in the first denoising 3DGS model whose difference exceeds the preset RGB threshold are removed, and the resulting color denoising 3DGS model is the second denoising 3DGS model.

9. A denoising system for a 3DGS model of a target object, characterized in that, It includes a preprocessing module, a sparse point cloud module, a dense point cloud module, a dense point cloud filtering module, a 3DGS model module, and a 3DGS model denoising module; The preprocessing module acquires original images of the target object from multiple perspectives, obtains a set of original images, and performs preprocessing to obtain a preprocessed image set. The complete image of the target object is composed of all the images in the original image set. The sparse point cloud module performs sparse reconstruction processing based on the preprocessed image set to obtain a sparse point cloud dataset. The dense point cloud module performs dense reconstruction processing on the sparse point cloud dataset to obtain a dense point cloud dataset. The dense point cloud filtering module performs scale restoration and pose correction processing on the dense point cloud dataset to obtain a normalized dense point cloud dataset; and performs filtering processing on the normalized dense point cloud dataset based on HSV color space thresholds to obtain a filtered dense point cloud dataset. The 3DGS model module performs Gaussian optimization and dense reconstruction on the sparse point cloud dataset to obtain a Gaussian dense point cloud dataset. Specifically, the sparse point cloud dataset is initialized based on a Gaussian distribution and optimized based on the principle of minimizing prediction error to obtain an optimized Gaussian sparse point cloud dataset; the optimized Gaussian sparse point cloud dataset is then densely reconstructed and optimized again based on the principle of minimizing prediction error to obtain a Gaussian dense point cloud dataset; based on the Gaussian dense point cloud dataset, combined with 3DGS reconstruction technology, a 3DGS model is obtained, and thus a 3DGS model point cloud dataset is obtained. The 3DGS model denoising module calculates the distance between corresponding 3D points in the filtered dense point cloud dataset and the 3DGS model point cloud dataset, removes 3D points in the 3DGS model whose distance is greater than a preset distance threshold, and obtains a denoised 3DGS model.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

11. A denoising apparatus for a 3DGS model of a target object, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images