Indoor furniture three-dimensional reconstruction and real-time editing method, system and equipment, medium and product

Abstract

The invention discloses an indoor furniture three-dimensional reconstruction and real-time editing method, system and device, a medium and a product, and relates to the technical field of three-dimensional reconstruction, and the method comprises the steps: obtaining furniture image data shot at multiple angles in real time; obtaining a camera internal parameter matrix, a camera external parameter matrix, a sparse point cloud and an RGB furniture image based on the furniture image data; based on the camera internal parameter matrix, the camera external parameter matrix and the RGB furniture image, rendering the sparse point cloud and performing iterative training according to a set number of iterations to obtain a three-dimensional Gaussian model; and performing real-time editing processing on the three-dimensional Gaussian model in the initial scene based on a set requirement to obtain a home decoration scene meeting the set requirement. The rendering quality, the training speed and the rendering speed of the three-dimensional Gaussian model can be optimized, the use difficulty is low, the placement of the furniture position can be edited in real time, the scene detail reduction degree is improved, and a user can check the edited result in real time.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a method, system, device, medium and product for 3D reconstruction and real-time editing of indoor furniture. Background Technology

[0002] The goal of 3D reconstruction is to convert a collection of images or videos of a scene into a 3D model that can be processed and understood by a computer. Early 3D reconstruction work did not involve deep learning, resulting in poor modeling quality when faced with complex lighting scenes.

[0003] Multi-view stereo (MVS) is a traditional method in 3D reconstruction technology. It relies on images captured from multiple different viewpoints and uses geometric relationships and triangulation methods to reconstruct 3D structures. The basic idea of ​​MVS is to find common object feature points in multi-view images through image registration and feature matching, calculate the positions of these points in 3D space, and then generate a 3D geometric model of the target. Its advantage lies in the mature theoretical basis of its geometric methods, which can generate high-resolution geometric structures. It is suitable for high-precision reconstruction of static scenes and is therefore widely used in industrial measurement, building scanning, and other scenarios. However, MVS has high requirements for the number of viewpoints and usually requires rich viewpoint information to accurately restore the depth information of objects. At the same time, it has high computational complexity, especially when the image resolution or the number of viewpoints is large, the overhead of feature matching and geometric calculation will increase significantly, limiting its use in large-scale scene reconstruction or real-time applications.

[0004] Neural Radiance Fields (NeRF) is a deep learning-based 3D reconstruction technique that has emerged in recent years. NeRF uses a neural network to map each 3D location to RGB values ​​and volume density, thereby generating viewpoint-dependent 2D images. By training the neural network on densely sampled points, NeRF learns implicit representations to describe the structure and lighting of a 3D scene. Thanks to the powerful expressive capabilities of neural networks, NeRF performs exceptionally well under complex lighting conditions and scene occlusion, generating high-fidelity rendering effects. Therefore, NeRF is particularly suitable for scene reconstruction applications that require high realism, such as high-precision rendering in virtual reality. However, due to the complexity of the model, NeRF has high computational costs, especially requiring a large number of parameters and long training times. Viewpoint generation requires network inference, making it difficult to implement efficiently in real-time applications. Furthermore, NeRF has high storage overhead, making it suitable for fine reconstruction of static scenes, but its performance is limited in dynamic scenes.

[0005] 3D Gaussian Splatting (3DGS) is a 3D reconstruction method that approximates object surfaces and lighting based on Gaussian ellipsoids distributed in 3D space. 3DGS represents a scene as multiple Gaussian points with different densities, sizes, and colors to approximate shapes and lighting distributions in 3D space. Through the sparse representation of Gaussian ellipsoids, 3DGS can efficiently generate viewpoints and has good storage efficiency, making it suitable for applications with high real-time rendering requirements, such as virtual reality and augmented reality. Compared to NeRF, 3DGS offers stronger real-time performance, relatively lower computational overhead, and reduced demands on video memory and storage. However, due to its reliance on point clouds and Gaussian voxel approximation, detail representation is somewhat limited, making it difficult to render high-precision surface details in complex geometries. 3DGS also has relatively limited performance for dynamic scenes or fast-moving objects.

[0006] The main execution flow of 3D Gaussian rendering technology is as follows: For the object to be modeled, the user takes pictures of the object from multiple perspectives using a mobile phone or other shooting device, obtaining about 100 RGB images, which are then input into the Structure from Motion (Sfm) algorithm. The Sfm algorithm first extracts features from the images, then matches the extracted features, and then uses the Bundle Adjustment (BA) algorithm to finally obtain a sparse point cloud, while estimating the camera pose during the shooting process. The purpose of the BA algorithm is to optimize the camera pose (also known as camera extrinsic parameters, i.e., the coordinates and orientation of the camera when taking the picture). The coordinates of each point in the sparse point cloud, the RGB values ​​corresponding to the initial values ​​of each Gaussian ellipsoid, and then a viewpoint is randomly selected. The alpha-blending algorithm is used to render the predicted image, and the loss is calculated with the real image under that viewpoint before training the Gaussian parameters. Among them, the Gaussian ellipsoid coordinates, spherical harmonic coefficients SH (used to describe color), transparency, Gaussian ellipsoid size, and Gaussian ellipsoid shape (controlled by the covariance matrix) all participate in the optimization. After multiple rounds of training, the 3D model of the object is described by a large set of Gaussian ellipsoids.

[0007] The earliest proposed 3DGS algorithm had obvious drawbacks: the rendered image was blurry, the geometric plane reconstruction was difficult, and the storage cost of the three-dimensional Gaussian model was high.

[0008] 3DGS explicitly models scenes using a large number of 3D ellipsoids, achieving faster training speeds compared to NeRF and enabling real-time new-viewpoint compositing rendering. Rendering quality is one of the primary research areas. Ajazziness, artifacts, and noise exist in 3DGS at different angles and in different scenes, resulting in poor image quality. In the inventors' known research, when the camera is far from the object or when rendering low-resolution images, pixels smaller than the size of the Gaussian ellipsoid cause noticeable distortion and jagged edges in the rendered image. Therefore, different sizes of Gaussians are used to describe the scene: large and coarse Gaussians for low resolution and small and precise Gaussians for high resolution. The large and coarse Gaussians are aggregated from the small and precise Gaussians. This dynamic aggregation strategy allows Gaussians to efficiently encode high and low frequency information, improving rendering quality at different resolutions while also increasing rendering speed to some extent. Similarly, in another known study, 3DGS lacks three-dimensional frequency constraints and two-dimensional dilation filters. For example, shrinking the image causes the Gaussian projection in pixel space to become smaller, easily leading to artifacts; conversely, scaling up causes the Gaussian projection to become larger, resulting in errors in the gaps between Gaussians. To eliminate artifacts during magnification, the authors applied regularization constraints to the 3D Gaussian during training, adding a low-frequency filter and a maximum Gaussian volume constraint. For artifacts in downsized images, a 2D box filter was added to address the dilation effect during downsizing. Furthermore, in another known study, the 3DGS compaction method suffered from over-reconstruction, where areas with large image standard deviations were covered with a small amount of large Gaussian material, resulting in significant blurring and artifacts in the rendering. This study proposes frequency annealing to achieve progressive frequency regularization. Specifically, it employs a Gaussian compaction process from large to small, gradually annealing and regularizing to high-frequency signals based on the principle of encoding large-scale low-frequency signals. This modeling of small-scale features minimizes the difference between the rendered image spectrum and the real image spectrum, ultimately improving image quality.

[0009] Point cloud densification (PCD) makes the originally sparse Gaussian point cloud denser from the perspective of point cloud initialization. PCD uses a pre-trained transformer to predict a single-channel depth map through RGB, and uses the depth map generated by scene RGB to build a dense point cloud. The dense point cloud is combined with the sparse point cloud generated by colmap to obtain the final initialized 3DGS point cloud. Combined with the camera position and angle sampling synthesis method proposed by the authors, the average PSNR (Peak Signal-to-Noise Ratio) reaches 20.48 and 20.55 on the ScanNet dataset and LLFF (Local Light Field Fusion) dataset, respectively. The SSIM (Structural Similarity) reaches 0.73 and 0.73, respectively, and the LPIPS (Learned Perceptual Image Patch Similarity) reaches 0.37 and 0.16, respectively. Another inventor's known research, inspired by MVS, utilizes prior knowledge of existing reconstructed geometry in the scene and patch matching to generate a new Gaussian function with accurate position and orientation, thereby achieving the goal of densifying the Gaussian initial point cloud.

[0010] Geometric quality is also a significant research area. Most indoor furniture has sharp edges and corners, and is primarily flat with few texture features. The three-dimensionality of a 3D Gaussian ellipsoid makes it difficult to accurately render a 2D plane. In one known study, the inventors jointly optimized the NeuS (Neural Implicit Surface Reconstruction) network and a 3D Gaussian sphere, using the 3D Gaussian sphere as a constraint for NeuS. This resulted in a high-quality reconstructed object surface mesh, but the training time was approximately ten times longer than that of the original 3DGS. Another study, considering the disorder inherent in the 3D Gaussian sphere, proposed that it needs to be aligned with the object surface to better represent scene geometry. Specifically, additional constraints are added to the 3D Gaussian sphere to ensure limited overlap between spheres and uniform distribution on the geometric surface; the spheres tend to be completely opaque or completely transparent; and they tend to be flat, with one of the x, y, and z components close to zero. This resulted in a scene with more accurate geometric features, and the proposed algorithm can extract an accurate scene mesh representation from the 3D Gaussian sphere within minutes. Unlike the two geometry quality optimization schemes mentioned above, which are still based on 3D Gaussian, 2DGS (2D Gaussian Splatting) is proposed to directly optimize the 2D Gaussian ellipsoid, as the rendering depth of 3D Gaussian varies across different viewpoints. This provides better multi-view consistency compared to 3D Gaussian. Based on a differentiable 3D Gaussian renderer, this research innovatively proposes a 2D Gaussian renderer. By utilizing 2D surface modeling, the intersection of light rays with the Gaussian ellipsoid, and volume integrals, it achieves viewpoint-consistent rendering and improves the quality of geometry reconstruction.

[0011] The number of Gaussian spheres is the most significant factor affecting training and rendering speed, and much research has focused on reducing it. One researcher calculated importance scores using opacity and Gaussian sphere size, pruned the Gaussian spheres based on these scores, and iterated multiple times to restore performance before pruning. Simultaneously, they reduced the spherical harmonic coefficients from third to second order and used vector quantization, compressing the Gaussian model to 1 / 15. Another known study employed a different pruning strategy, introducing a one-dimensional 0-1 learnable mask to the Gaussian spheres, which applied to the scaling and opacity coefficients of each sphere. When the mask = 0, it was equivalent to pruning. The sigmoid average of the mask was also included in the model loss calculation to limit the degree of pruning, ultimately reducing storage space requirements to approximately 1 / 20. Due to the mask's involvement, training time was extended by about 1 / 4. In the research that proposed 3DGS, an improved Gaussian sphere clipping algorithm and spherical harmonic function coefficient compression algorithm were developed. The pixel spatial projection redundancy was calculated, and 3D Gaussians with high redundancy were removed proportionally. For the storage of spherical harmonic function coefficients, two cases were considered: when the standard deviation of the color expressed by a Gaussian sphere under different viewpoints is lower than a set threshold, only the basic RGB values ​​are saved; when the color difference expressed by a Gaussian sphere using low-order and high-order coefficients is lower than a set threshold, only the low-order coefficients are retained. Besides position parameters, vector quantization techniques were used for other parameters such as the scaling, rotation matrix, and spherical harmonic function coefficients of the Gaussian sphere, reducing storage space requirements to 1 / 25. Another known study by the inventors adopted a more aggressive clipping strategy while maintaining rendering quality consistent with 3DGS. Importance was calculated based on parameters such as transparency and the Gaussian projection area. Furthermore, for each ray (pixel), the Gaussian that contributes the most to its color was retained. The training speed was twice that of 3DGS, while the rendering speed on the Mip-NERF dataset was 1 / 3 faster than Reduced3DGS.

[0012] Real-time scene editing serves as the user-end application of this project. Users select desired furniture, place it within the indoor scene, and receive real-time rendering results. The inventors' known research is the first end-to-end real-time interactive 3D Gaussian scene editing system, achieving high-quality 3D semantic segmentation through two-stage 3D anchor boxes to obtain the associated 3D Gaussian coordinates for each entity. After users complete translation, rotation, and other editing operations, the authors propose a 2D image inpainting tool and a Gaussian reprojection-based inpainting method, followed by minor fine-tuning to complete a scene editing cycle.

[0013] In summary, existing research focuses on optimizing specific details of the 3D Gaussian reconstruction algorithm, rather than addressing the entire end-to-end user experience and prioritizing ease of use for actual users. This results in a high learning curve, low system smoothness, and an inability to meet the higher requirements of home decoration scenarios in terms of scene detail reproduction. Summary of the Invention

[0014] The purpose of this application is to provide a method, system, device, medium, and product for 3D reconstruction and real-time editing of indoor furniture, which optimizes the rendering quality, training speed, and rendering speed of 3D Gaussian models, is easy to use, enables real-time editing of furniture placement, improves scene detail reproduction, and allows users to view the edited results in real time.

[0015] To achieve the above objectives, this application provides the following solution:

[0016] Firstly, this application provides a method for three-dimensional reconstruction and real-time editing of indoor furniture, including:

[0017] Real-time acquisition of furniture image data captured from multiple angles;

[0018] Based on the furniture image data, camera intrinsic parameter matrix, camera extrinsic parameter matrix, sparse point cloud and RGB furniture image are obtained;

[0019] Based on the camera intrinsic matrix, the camera extrinsic matrix, and the RGB furniture image, the sparse point cloud is rendered and iteratively trained according to a set number of iterations to obtain a three-dimensional Gaussian model.

[0020] Based on the set requirements, the 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

[0021] Optionally, based on the furniture image data, camera intrinsic matrix, camera extrinsic matrix, sparse point cloud, and RGB furniture image are obtained, including:

[0022] The furniture image data is downsampled to obtain a downsampled RGB furniture image;

[0023] The downsampled RGB furniture image is sequentially subjected to feature extraction, feature matching, point cloud generation, and parameter optimization to obtain the camera intrinsic matrix, camera extrinsic matrix, and initial sparse point cloud.

[0024] The initial sparse point cloud is registered based on the camera extrinsic matrix, and the registered sparse point cloud is filtered to obtain the sparse point cloud.

[0025] The RGB furniture image is obtained by removing environmental information from the downsampled RGB furniture image.

[0026] Optionally, the process of sequentially performing feature extraction, feature matching, and point cloud generation and parameter optimization on the downsampled RGB furniture image to obtain the camera intrinsic parameter matrix, camera extrinsic parameter matrix, and initial sparse point cloud includes:

[0027] Feature extraction is performed on the downsampled RGB furniture images to obtain a camera intrinsic parameter matrix and a feature vector set; the feature vector set includes m×n feature vectors; n is the number of feature points obtained by feature extraction on each downsampled RGB furniture image; the feature points are represented as feature vectors; m is the number of downsampled RGB furniture images.

[0028] An exhaustive matching method is adopted. Based on the feature points of the downsampled RGB furniture images, each downsampled RGB furniture image is matched with other downsampled RGB furniture images to obtain a set of matching feature points. This includes: determining the cosine similarity value between feature points of every two downsampled RGB furniture images based on the feature vectors corresponding to the feature points of the downsampled RGB furniture images; using the feature points that meet the set similarity value as the matching feature points of the two downsampled RGB furniture images, and constructing a set of matching feature points based on the matching feature points of every two downsampled RGB furniture images.

[0029] Based on the set of matching feature points, the two downsampled RGB furniture images with the most matching feature points are selected, and the matching feature points are sorted in descending order based on the cosine similarity value of all matching feature points of the two downsampled RGB furniture images to obtain the matching feature point sequence.

[0030] The first p matching feature points in the matching feature point sequence are selected to determine the initial matrix of camera extrinsic parameters;

[0031] An initial point cloud is generated based on the matching feature points of the two downsampled RGB furniture images with the most matching feature points. Using the BA optimization algorithm, the downsampled RGB furniture images with the most matching feature points from the set of matching feature points are selected sequentially. Based on the matching feature points and corresponding feature vectors of these downsampled RGB furniture images, the initial point cloud and the initial camera extrinsic matrix are optimized. This process continues until all downsampled RGB furniture images have been optimized for both the initial point cloud and the initial camera extrinsic matrix. Finally, BA optimization is performed on the optimized initial point cloud and the optimized initial camera extrinsic matrix to obtain the camera extrinsic matrix and the initial sparse point cloud.

[0032] Optionally, based on the camera intrinsic matrix, the camera extrinsic matrix, and the RGB furniture image, the sparse point cloud is rendered and iteratively trained according to a set number of iterations to obtain a 3D Gaussian model, including:

[0033] An initial three-dimensional Gaussian model is constructed based on the sparse point cloud;

[0034] The point cloud in the RGB furniture image corresponding to the camera intrinsic parameter matrix is ​​obtained from the sparse point cloud, and the point cloud in the RGB furniture image is rendered using the volume rendering equation to obtain the rendered initial three-dimensional Gaussian model.

[0035] Based on the RGB furniture image, the backpropagation algorithm is used to iteratively train the rendered initial 3D Gaussian model according to a set number of iterations until the training reaches the set number of iterations. The trained initial 3D Gaussian model is then used as the 3D Gaussian model.

[0036] Optionally, the 3D Gaussian model is edited in real time in the initial scene based on set requirements to obtain a home decoration scene that meets the set requirements, including:

[0037] Based on the set requirements, the 3D Gaussian model is loaded into the initial scene and rendered and visualized.

[0038] Based on the set requirements, the rendered and visualized 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

[0039] Optionally, the means of real-time editing of the three-dimensional Gaussian model include: translating, rotating and / or scaling the three-dimensional Gaussian model.

[0040] Secondly, this application provides a system for three-dimensional reconstruction and real-time editing of indoor furniture, including:

[0041] The data acquisition module is used to acquire furniture image data from multiple angles in real time;

[0042] The data preprocessing module is used to obtain the camera intrinsic parameter matrix, camera extrinsic parameter matrix, sparse point cloud, and RGB furniture image based on the furniture image data;

[0043] The pixel rendering module is used to render the sparse point cloud based on the camera intrinsic parameter matrix, the camera extrinsic parameter matrix and the RGB furniture image, and to perform iterative training according to a set number of iterations to obtain a three-dimensional Gaussian model.

[0044] The visual editing module is used to perform real-time editing of the three-dimensional Gaussian model in the initial scene based on set requirements, so as to obtain a home decoration scene that meets the set requirements.

[0045] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the indoor furniture three-dimensional reconstruction and real-time editing method described above.

[0046] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional reconstruction and real-time editing method for indoor furniture described above.

[0047] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the three-dimensional reconstruction and real-time editing method for indoor furniture described above.

[0048] According to the specific embodiments provided in this application, this application has the following technical effects:

[0049] This application provides a method, system, device, medium, and product for 3D reconstruction and real-time editing of indoor furniture. It reduces user difficulty by acquiring furniture image data from multiple angles in real time. Based on the furniture image data, it obtains camera intrinsic matrix, camera extrinsic matrix, sparse point cloud, and RGB furniture image. Based on the camera intrinsic matrix, camera extrinsic matrix, and RGB furniture image, it renders the sparse point cloud and iteratively trains it according to a set number of iterations to obtain a 3D Gaussian model. It optimizes the rendering quality, training speed, and rendering speed of the 3D Gaussian model, improving the scene detail reproduction. Based on set requirements, it performs real-time editing of the 3D Gaussian model in the initial scene to obtain a home decoration scene that meets the set requirements. It realizes an end-to-end user workflow and allows users to edit the furniture placement in real time, and view the edited results immediately. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a method for three-dimensional reconstruction and real-time editing of indoor furniture in one embodiment of this application;

[0052] Figure 2 This is a schematic diagram of a method for three-dimensional reconstruction and real-time editing of indoor furniture provided in another embodiment of this application;

[0053] Figure 3 This is a schematic diagram illustrating a method for capturing furniture image data according to an embodiment of this application;

[0054] Figure 4 This is a photographic illustration of a piece of furniture provided in one embodiment of this application;

[0055] Figure 5 This is a schematic diagram of a visual editing interface provided in one embodiment of this application;

[0056] Figure 6 This is a schematic diagram of a home decoration scene provided in an embodiment of this application;

[0057] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] Lightweighting the system is essential for the widespread application of 3D Gaussian rendering. Considering that this application involves 3D reconstruction and real-time rendering of interior furniture, the inventors found that most research indicates that training and rendering of a single interior scene only requires a single RTX 3090 graphics card to achieve 150 FPS. Users are likely to edit interior scenes on mobile devices (such as computers), thus limiting rendering speed and the size of the 3D Gaussian model. Furthermore, it is necessary to reduce the computational power required for modeling to quickly add all furniture to the model database. Given that the number of Gaussian spheres is the most significant factor affecting training and rendering speed, existing research typically focuses on reducing the number of Gaussian spheres.

[0061] In one exemplary embodiment, such as Figure 1 As shown, a method for 3D reconstruction and real-time editing of indoor furniture is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is used to edit a home decoration scene that meets set requirements, as an example, including:

[0062] Step 100: Acquire furniture image data from multiple angles in real time.

[0063] Step 200: Obtain the camera intrinsic matrix, camera extrinsic matrix, sparse point cloud, and RGB furniture image based on the furniture image data.

[0064] Step 300: Based on the camera intrinsic matrix, camera extrinsic matrix and RGB furniture image, render the sparse point cloud and iteratively train it according to the set number of iterations to obtain a 3D Gaussian model.

[0065] Step 400: Based on the set requirements, the 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

[0066] As an optional implementation, to improve the accuracy of the obtained camera intrinsic matrix, camera extrinsic matrix, sparse point cloud, and RGB furniture image, step 200 includes:

[0067] Step 210: Downsample the furniture image data to obtain a downsampled RGB furniture image.

[0068] Step 220: The downsampled RGB furniture image is subjected to feature extraction, feature matching, point cloud generation and parameter optimization in sequence to obtain the camera intrinsic parameter matrix, camera extrinsic parameter matrix and initial sparse point cloud.

[0069] Step 230: Register the initial sparse point cloud based on the camera extrinsic matrix, and filter the registered sparse point cloud to obtain the sparse point cloud.

[0070] Step 240: Remove environmental information from the downsampled RGB furniture image to obtain the RGB furniture image.

[0071] Step 220 includes:

[0072] Step 221: Extract features from the downsampled RGB furniture images to obtain the camera intrinsic parameter matrix and feature vector set. The feature vector set includes m×n feature vectors. n is the number of feature points obtained from feature extraction of each downsampled RGB furniture image. Feature points are represented as feature vectors. m is the number of downsampled RGB furniture images.

[0073] Step 222: Using an exhaustive matching method, based on the feature points of the downsampled RGB furniture images, each downsampled RGB furniture image is matched with other downsampled RGB furniture images to obtain a set of matching feature points. Specifically, this includes: determining the cosine similarity value between feature points of every two downsampled RGB furniture images based on the feature vectors corresponding to the feature points of the downsampled RGB furniture images. Feature points that meet the set similarity value are used as matching feature points between the two downsampled RGB furniture images, and a set of matching feature points is constructed based on the matching feature points of every two downsampled RGB furniture images.

[0074] Step 223: Based on the matching feature point set, select the two downsampled RGB furniture images with the most matching feature points, and sort the matching feature points in descending order based on the cosine similarity values ​​of all matching feature points in the two downsampled RGB furniture images to obtain a matching feature point sequence. Select the first p matching feature points in the matching feature point sequence to determine the initial matrix of camera extrinsic parameters.

[0075] Step 224: Generate an initial point cloud based on the matching feature points of the two downsampled RGB furniture images with the most matching feature points. Using the BA optimization algorithm, sequentially select the downsampled RGB furniture image with the most matching feature points from the set of matching feature points of these two downsampled RGB furniture images. Based on the matching feature points and corresponding feature vectors of this downsampled RGB furniture image, optimize the initial point cloud and the initial camera extrinsic matrix. After optimizing the initial point cloud and the initial camera extrinsic matrix for all downsampled RGB furniture images, perform BA optimization on the optimized initial point cloud and the optimized initial camera extrinsic matrix to obtain the camera extrinsic matrix and the initial sparse point cloud.

[0076] For example, the process of obtaining camera intrinsic matrix, camera extrinsic matrix, sparse point cloud and RGB furniture image based on furniture image data includes coarse processing, point cloud planar registration and filtering and image masking.

[0077] The initial coarse processing involves adjusting the resolution of the furniture image data by downsampling it to obtain a downsampled RGB furniture image. In practical applications, downsampling the furniture image data to below 1920*1080 is optimal. Then, the open-source incremental SFM algorithm is used to sequentially perform feature extraction, feature matching, point cloud generation, and parameter optimization on the downsampled RGB furniture image, resulting in the camera intrinsic matrix, camera extrinsic matrix, and initial sparse point cloud.

[0078] The process of feature extraction, feature matching, point cloud generation, and parameter optimization is as follows:

[0079] Feature Extraction: The captured furniture image data typically includes an EXIF ​​file, from which the camera intrinsic parameter matrix can be directly extracted. Scale-invariant feature transform (SIFT) is used to extract features from each downsampled RGB furniture image, resulting in a feature vector set. The feature vector set consists of m×n feature vectors. n represents the number of feature points obtained from feature extraction on each downsampled RGB furniture image. Each feature point is represented by a feature vector. m represents the number of downsampled RGB furniture images. In this embodiment, the feature vector is 128-dimensional, with each dimension representing the gradient information of the feature point.

[0080] Feature matching: An exhaustive matching method is used. Each downsampled RGB furniture image is matched once with every other downsampled RGB furniture image. Based on the feature vectors corresponding to the feature points of each image in the feature vector set obtained from feature extraction, feature matching is performed between every two downsampled RGB furniture images. Specifically, based on the feature vectors obtained from feature extraction, the cosine similarity value between the feature points of every two downsampled RGB furniture images is determined. Feature points that meet the set similarity value are used as matching feature points between these two downsampled RGB furniture images. After all downsampled RGB furniture images have been matched, a matching feature point set is constructed based on the matching feature points between every two downsampled RGB furniture images. Based on the matching feature point set, the two downsampled RGB furniture images with the most matching feature points are selected. The matching feature points are then sorted in descending order based on the cosine similarity value of all matching feature points between the two downsampled RGB furniture images, resulting in a matching feature point sequence. The first p matching feature points in the matching feature point sequence are selected to determine the initial matrix of camera extrinsic parameters.

[0081] In practical applications, the exhaustive matching method involves creating an image pair for each image with every other image. It iterates through each image pair and determines the cosine similarity value of all feature points between each image pair based on the feature vectors of the feature points in each image during feature extraction. Then, based on the cosine similarity value and a predetermined similarity value, matching feature points between the two images in that image pair are determined. This process yields matching feature points for every pair of images, forming a set of matching feature points. The image pair with the most matching feature points is selected from this set, and the matching feature points are sorted in descending order based on the cosine similarity value of the matching feature points between the two images in the pair, resulting in a sequence of matching feature points. The first eight matching feature points in this sequence can be used to determine the initial matrix of camera extrinsic parameters.

[0082] Point Cloud Generation and Parameter Optimization: The goal of point cloud generation and parameter optimization is to gradually increase the viewing angle, iteratively optimize the reprojection error, further accurately calculate the camera extrinsic matrix, and determine the visibility of each point in the point cloud under different viewing angles. First, an initial point cloud is generated based on the matching feature points of the two downsampled RGB furniture images with the most matching feature points. Then, using the Base Algorithm (BA) optimization algorithm, the downsampled RGB furniture image with the highest sum of matching feature points between its set and the two downsampled RGB furniture images is selected sequentially. The matching feature points between this image and the two images are triangulated, and the processed matching feature points are added to the initial point cloud as new point clouds. Simultaneously, a new camera extrinsic matrix is ​​determined based on the matching feature points between this image and the two images, as well as the initial camera extrinsic matrix. Another BA optimization is then performed to improve the accuracy of the newly added point cloud and the new camera extrinsic matrix, thereby minimizing the reprojection error. After adding and optimizing the matching feature points between all downsampled RGB furniture images in the matching feature point set and these two downsampled RGB furniture images, a global BA optimization is finally performed on the optimized initial point cloud and the optimized initial camera extrinsic matrix to obtain a more accurate camera extrinsic matrix and initial sparse point cloud.

[0083] Point cloud planar registration and filtering involves registering the initial sparse point cloud based on the camera extrinsic matrix and then filtering the registered sparse point cloud to obtain the final sparse point cloud. First, coarse registration is performed on the initial sparse point cloud using the camera extrinsic matrix, and then the point cloud coordinate system of the initial sparse point cloud is affinely transformed to the world coordinate system.

[0084] Because the camera extrinsic matrix is ​​estimated using an open-source incremental SFM algorithm rather than being a completely precise value, the transformed point cloud coordinate system may still have a certain angle with the world coordinate system. Therefore, the furniture model (i.e., the 3D Gaussian model) imported during real-time editing will not be perfectly parallel to the xy plane of the world coordinate system. In acquiring furniture image data from multiple angles, the ground in the actual shooting environment is the largest plane in the initial sparse point cloud. Utilizing this characteristic, the SACSegmentation algorithm in the Point Cloud Processing Library (PCL) is called to segment out the largest plane in the initial sparse point cloud. Based on this, the normal vector n of the largest plane in the initial sparse point cloud is determined. f and the normal vector n of the xy plane of the world coordinate system. xy Perform a cross product to obtain the rotation axis r: r = n f ×n xy Then, using the formula θ=-atan2(||r||,n f ·n xy Calculate the rotation angle θ. Finally, rotate the initial sparse point cloud and the camera extrinsic matrix using the rotation axis and the angle to accurately register the point cloud.

[0085] Since there is a one-to-one correspondence between the initial points of the 3D Gaussian model and the points in the sparse point cloud (i.e., the sparse point cloud is the initial point cloud of the 3D Gaussian model, and the position (xyz) and color (rgb) of the point cloud are used as the initial values ​​of the 3D Gaussian points), the order should be: registration -> filtering -> forming a sparse point cloud. Filtering can reduce the number of points unrelated to the target modeling object (i.e., furniture), reduce the amount of computation during the training of the 3D Gaussian model, and speed up the training process). A large number of points unrelated to the target modeling object will greatly increase the amount of invalid computation. Therefore, the Pass-Through Filter and RadiusOutlierRemoval provided by the PCL library are used to filter the initial sparse point cloud after registration. The former sets the bounding box and filters points outside the box, while the latter sets the minimum number of neighboring points for each point and filters points with insufficient neighboring points, resulting in the filtered initial sparse point cloud (i.e., the sparse point cloud). The sparse point cloud is then used as the initial point cloud of the 3D Gaussian model.

[0086] Image masking is used to remove environmental information from downsampled RGB furniture images, resulting in RGB furniture images. The SAM2-S model is used to label each downsampled RGB furniture image with an object mask, removing environmental information (e.g., setting the environmental information to black), thus obtaining the RGB furniture image.

[0087] Step 300 includes:

[0088] Step 310: Construct an initial 3D Gaussian model based on the sparse point cloud.

[0089] Step 320: Obtain the point cloud in the RGB furniture image corresponding to the camera intrinsic parameter matrix from the sparse point cloud, and render the point cloud in the RGB furniture image using the volume rendering equation to obtain the rendered initial 3D Gaussian model.

[0090] Step 330: Based on the RGB furniture image, the backpropagation algorithm is used to iteratively train the rendered initial 3D Gaussian model according to the set number of iterations until the training reaches the set number of iterations, and the trained initial 3D Gaussian model is used as the 3D Gaussian model.

[0091] For example, an initial 3D Gaussian model is constructed based on sparse point clouds. Based on the RGB furniture image, camera intrinsic and extrinsic matrices, a known viewpoint (i.e., the angle corresponding to the RGB furniture image) is randomly selected. Gaussian points within the view frustum at this viewpoint (i.e., the point cloud corresponding to the RGB furniture image) are obtained. The color of each pixel in the sparse point cloud is rendered from near to far using volume rendering equations. The RGB furniture image obtained through image masking is used as supervision. Backpropagation optimizes the parameters of the 3D Gaussian ellipsoid, including position (XYZ), size (Scale), shape (controlled by the covariance matrix ∑), color (controlled by the spherical harmonic function coefficients SH), and transparency (alpha), to obtain the rendered initial 3D Gaussian model. Based on the RGB furniture image, the backpropagation algorithm is used to iteratively train the rendered initial 3D Gaussian model according to a set number of iterations until the set number of iterations is reached. The trained initial 3D Gaussian model is then used as the 3D Gaussian model. It should be noted that in this embodiment, the number of iterations can be 30,000, and the 3D Gaussian model can be stored as a PLY format file.

[0092] Step 400 includes:

[0093] Step 410: Load a 3D Gaussian model into the initial scene based on the set requirements and perform rendering and visualization processing.

[0094] Step 420: Based on the set requirements, the rendered and visualized 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

[0095] The methods for real-time editing of 3D Gaussian models include: translating, rotating, and / or scaling the 3D Gaussian model.

[0096] In one exemplary embodiment, such as Figure 2 As shown, in conjunction with the steps in the above embodiments, a method for 3D reconstruction and real-time editing of indoor furniture is used to perform 3D reconstruction and real-time editing of a certain piece of furniture.

[0097] Step S1, as follows Figure 3 As shown, the system acquires real-time image data of a piece of furniture from multiple angles. Each image should ideally contain the entire piece of furniture, and at least three images from different perspectives should be captured of a single location on the furniture. For example, a particular side of the furniture should be included in at least three images from different perspectives. Figure 4 As shown. Furniture should ideally have texture. If the furniture itself lacks texture, the shooting environment or background needs to have obvious texture features. Sufficient lighting is required during the shooting process. Avoid white walls, mirror reflections, perspective objects, and high-contrast scenes. For a single piece of furniture, images should be taken in the same environment. For furniture with a volume of less than 1 cubic meter, approximately 100 images from different angles are needed. Some images of a particular piece of furniture are shown below. Figure 3 As shown.

[0098] Adjust the number and angle of photos according to actual needs and furniture size, but the lens should not be too close (only showing a part of the furniture) or too far (making it difficult to identify furniture features), and the number of furniture photos should be within 200 and the number of scene photos should be within 400.

[0099] Step S2, the process of obtaining the camera intrinsic matrix, camera extrinsic matrix, sparse point cloud and RGB furniture image based on furniture image data, includes coarse processing, point cloud planar registration and filtering, and image masking.

[0100] The initial processing involves adjusting the image resolution, with downsampling to below 1920*1080 being optimal. Then, using the downsampled images, the open-source incremental SFM algorithm is applied to perform image feature extraction, image feature matching, point cloud generation, and parameter optimization to obtain the initial sparse point cloud, camera intrinsic parameter matrix, and camera extrinsic parameter matrix for each image.

[0101] The initial sparse point cloud is sequentially registered and filtered to obtain the sparse point cloud.

[0102] The SAM2-S model is used to label each downsampled RGB image with an object mask, and the environmental information is removed (set to black) to obtain the RGB furniture image.

[0103] Step S3: Based on the camera intrinsic matrix, camera extrinsic matrix, and RGB furniture image, render the sparse point cloud and iteratively train it according to the set number of iterations to obtain a 3D Gaussian model. Construct an initial 3D Gaussian model based on the sparse point cloud. Using the obtained camera intrinsic matrix, camera extrinsic matrix, and RGB furniture image, randomly select a known viewpoint, obtain Gaussian points within the view frustum at that viewpoint, and render the color of each pixel from near to far using the volume rendering equation. Use the RGB furniture image obtained from image masking as supervision, and backpropagate to optimize the parameters in the model, including the position (XYZ), size (Scale), shape (controlled by the covariance matrix ∑), color (controlled by the spherical harmonic function coefficients SH), and transparency (alpha) of the 3D Gaussian ellipsoid. Iteratively train the initial 3D Gaussian model according to the set number of iterations, and store the iteratively trained initial 3D Gaussian model as the 3D Gaussian model in a ply format file.

[0104] Step S4 involves real-time editing of the 3D Gaussian model within the initial scene based on set requirements, resulting in a home decoration scene that meets those requirements. The visual editing interface is shown below. Figure 5As shown, the left-hand Load area is for selecting and loading 3D Gaussian models, while the right-hand area is for real-time rendering and editing. Search Filters (comma separated) are the search filters. Users can load / unload / replace selected and loaded 3D Gaussian models in real-time using the buttons on the left. Clicking the Add Scene button adds a 3D Gaussian model, and the Browse button browses the system directory (i.e., the directory where the 3D Gaussian models stored in ply format in step S3) and loads them. The path to the currently loaded model will be displayed at the end of the same line. Figure 5 The display method of the path is for illustrative purposes only and does not constitute a specific limitation on the content of this application. The Remove button is used to delete the corresponding 3D Gaussian model (the scene must have at least one 3D model, so the Remove buttons start from number 2). The Edit button is used to select the currently edited 3D model. Only one 3D model can be in edit mode at a time. When in edit mode, the selected model will appear reddish during rendering. In addition to the Edit button, you can also directly click on the model in the rendering area on the right to select the model to be edited. The Splitscreen button indicates split screen, and the Highlight Border button indicates a highlighted border. The Trans, Rotate, and scale buttons below provide three editing modes, corresponding to translation, rotation, and scaling, respectively. An affine transformation can also be performed on a 3D Gaussian model. The Clear button is used to exit edit mode and return to the camera adjustment mode. Adjust the perspective relative to the model or edit the model on the right. Based on the set requirements (i.e. user needs), a 3D Gaussian model is loaded into the initial scene and rendered and visualized. The rendered and visualized 3D Gaussian model is then edited in real time in the initial scene, allowing users to place furniture (i.e., the furniture corresponding to the 3D Gaussian model) in the target location to obtain a home decoration scene that meets the set requirements.

[0105] The methods for real-time editing of 3D Gaussian models include: translating, rotating and / or scaling the 3D Gaussian model, as well as 3D segmentation of the model and replacing the model from the scene.

[0106] For example, the editing target (i.e., the 3D Gaussian model that the user wants to edit) can be selected according to the set requirements.

[0107] The visual editing interface offers two methods for selecting the editing target. Users can select the model to be edited using the Edit 1 to Edit 3 buttons on the left, or select the editing target by clicking the editing and rendering interface on the right. When rendering the image, a matrix with the same dimensions as the image is stored, where W and H represent the rows and columns of the matrix, respectively, with a default value of 0 and an initial fullness T = 1. When using Gaussian g rendering, W...x H y The pixel at position (i.e., the Wth pixel in the matrix) x Line H y When the pixel corresponding to the column position is reached, next T =T*(1-g alpha ), to obtain the fullness of the next Gaussian g+1 rendering. T If the saturation change of the Gaussian is next T -T is greater than W in the matrix x H y The value at the position is used to replace the value in the matrix with the current change amount, and the current Gaussian number g is recorded. Then, by determining which model this Gaussian belongs to, the target editing model is determined. Simply put, the Gaussian that contributes the most to filling that pixel is recorded.

[0108] It can translate, rotate, and / or scale 3D Gaussian models.

[0109] 1) Translation

[0110] Translation only requires an affine transformation of the three-dimensional Gaussian coordinates XYZ. If the transformation amount is Δ using the keyboard / mouse, the three directional components are Δ... X Δ Y Δ Z The final translation transformation coordinates T of the three-dimensional Gaussian coordinates XYZ XYZ As shown in formula (1):

[0111]

[0112] 2) Rotation

[0113] The input rotation matrix is ​​a 3×3 R i The rotation requires adjustments to three parameters: the three-dimensional Gaussian coordinates (XYZ), the covariance matrix ∑, and the spherical harmonic coefficients SH. To ensure the object rotates about its own center of mass, the model's center of mass position needs to be recorded. First, the model's center of mass is translated to the origin, then the rotation transformation is performed, and finally, the reverse translation is performed to obtain the rotation result R about the XYZ. XYZ As shown in formula (2):

[0114] R XYZ =T XYZ ·R i ·(-T XYZ )·XYZ (2)

[0115] ∑ is represented by quaternions in Gaussian geometry, which requires the rotation matrix R to be used. i After converting to quaternions, multiply by left. This is done in two parts: converting the rotation matrix to quaternions and left multiplication. Starting from the rotation matrix R... i Obtain the quaternion R quatThe process is shown in formulas (3)-(8), where formula (3) extracts each value of the matrix. Formula (9) represents the left multiplication operation, yielding the rotation result R of the covariance. ∑ .

[0116] r xy =R i [x,y],x,y∈[0,2] (3)

[0117]

[0118] x=(r 21 -r 12 ) / (4*w) (5)

[0119] y = (r 02 -r 20 ) / (4*w) (6)

[0120] z=(r 10 -r 01 ) / (4*w) (7)

[0121] R quat =[w,x,y,z] (8)

[0122] R ∑ =R quat ·∑ (9)

[0123] In the formula, T XYZ Let r represent the coordinates of the three-dimensional Gaussian after translation transformation. xy Let [w, x, y, z] represent the elements in a 3×3 rotation matrix, where [w, x, y, z] represents the four values ​​of a quaternion.

[0124] The case of spherical harmonic coefficients SH is slightly more complex, with the rotation matrix being R. i For example, wigner_D matrices of different orders are generated as SH rotation matrices according to formulas (10)-(13). Formula (12) calls the matrix_to_angles function of the o3nn library to convert the rotation matrix into three rotation angles, which are stored in vector A. Formula (13) calls the wigner_D function of the o3nn library to generate the corresponding SH rotation matrix R from the rotation angle and order. deg Formula (14) is rotated over the three degrees of SH respectively to obtain the spherical harmonic function R after rotation. SH .

[0125]

[0126] R p =P -1 ·R i ·P (11)

[0127] A = matrix_to_angles(R) p (12)

[0128] R deg =wigner_D(deg,A[0],-A[1],A[2]),deg∈[1,2,3] (13)

[0129] R SH =R deg ·SH deg ,deg∈[1,2,3] (14)

[0130] In the formula, P is an intermediate variable used to construct the pre-matrix for the rotation angle of SH, and R... p The preceding matrix representing the rotation angle of SH, SH deg Represents spherical harmonic functions of order 1 to 3.

[0131] 3) Scaling

[0132] Scaling requires transforming two parameters: XYZ and scale. The three directional components are S... X ,S Y ,S Z Multiply the three components XYZ respectively, and then sum the scale by taking the log of the three components and the original scale. The scaling result is obtained by formula (15)-formula (16).

[0133]

[0134] In the formula, S XYZ S represents the XYZ coordinates of the scaled Gaussian point. scale This represents the scale value of the Gaussian after scaling.

[0135] Combining the aforementioned editing interface and the affine transformation techniques for 3D Gaussian models, users can load multiple 3D Gaussian models into the initial scene according to their needs. To conserve GPU memory resources, all 3D Gaussian models can be loaded into a model list in memory. The models in the list are then concatenated and copied to GPU memory for rendering using the 3D Gaussian rendering pipeline. When users perform editing operations on the models (viewpoint transformation, translation, rotation, scaling), the affine transformation matrix is ​​used to manipulate the corresponding 3D Gaussian models on the GPU. Models are only reloaded and concatenated from memory when the user adds, deletes, or replaces 3D Gaussian models. Figure 6 As shown, this demonstrates placing three chairs and a table in a garden. Further optimization of scene lighting and shadows could make the home decor scene more realistic.

[0136] Based on the same inventive concept, this application also provides a system for implementing the aforementioned method for 3D reconstruction and real-time editing of indoor furniture. The solution provided by this system is similar to the solution described in the above method; therefore, the specific limitations in the following embodiments of 3D reconstruction and real-time editing of indoor furniture can be found in the limitations of the method described above, and will not be repeated here.

[0137] In one exemplary embodiment, a system for 3D reconstruction and real-time editing of indoor furniture is provided, comprising:

[0138] The data acquisition module is used to acquire furniture image data from multiple angles in real time.

[0139] The data preprocessing module is used to obtain the camera intrinsic matrix, camera extrinsic matrix, sparse point cloud, and RGB furniture image based on the furniture image data.

[0140] The pixel rendering module is used to render sparse point clouds based on the camera intrinsic matrix, camera extrinsic matrix, and RGB furniture images, and to iteratively train the model according to a set number of iterations to obtain a 3D Gaussian model.

[0141] The visual editing module is used to perform real-time editing of the 3D Gaussian model in the initial scene based on the set requirements, so as to obtain a home decoration scene that meets the set requirements.

[0142] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related to the 3D reconstruction and real-time editing method for indoor furniture. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D reconstruction and real-time editing method for indoor furniture.

[0143] Those skilled in the art will understand that Figure 7The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0144] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0145] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0148] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0149] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0150] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for three-dimensional reconstruction and real-time editing of indoor furniture, characterized in that, include: Real-time acquisition of furniture image data captured from multiple angles; Based on the furniture image data, camera intrinsic parameter matrix, camera extrinsic parameter matrix, sparse point cloud and RGB furniture image are obtained; Based on the camera intrinsic matrix, the camera extrinsic matrix, and the RGB furniture image, the sparse point cloud is rendered and iteratively trained according to a set number of iterations to obtain a three-dimensional Gaussian model. Based on the set requirements, the 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

2. The method for three-dimensional reconstruction and real-time editing of indoor furniture according to claim 1, characterized in that, Based on the furniture image data, camera intrinsic parameter matrix, camera extrinsic parameter matrix, sparse point cloud, and RGB furniture image are obtained, including: The furniture image data is downsampled to obtain a downsampled RGB furniture image; The downsampled RGB furniture image is sequentially subjected to feature extraction, feature matching, point cloud generation, and parameter optimization to obtain the camera intrinsic matrix, camera extrinsic matrix, and initial sparse point cloud. The initial sparse point cloud is registered based on the camera extrinsic matrix, and the registered sparse point cloud is filtered to obtain the sparse point cloud. The RGB furniture image is obtained by removing environmental information from the downsampled RGB furniture image.

3. The method for three-dimensional reconstruction and real-time editing of indoor furniture according to claim 2, characterized in that, The process of sequentially performing feature extraction, feature matching, point cloud generation, and parameter optimization on the downsampled RGB furniture image to obtain the camera intrinsic matrix, camera extrinsic matrix, and initial sparse point cloud includes: Feature extraction is performed on the downsampled RGB furniture images to obtain a camera intrinsic parameter matrix and a feature vector set; the feature vector set includes m×n feature vectors; n is the number of feature points obtained by feature extraction on each downsampled RGB furniture image; the feature points are represented as feature vectors; m is the number of downsampled RGB furniture images. An exhaustive matching method is adopted. Based on the feature points of the downsampled RGB furniture images, each downsampled RGB furniture image is matched with other downsampled RGB furniture images to obtain a set of matching feature points. This includes: determining the cosine similarity value between feature points of every two downsampled RGB furniture images based on the feature vectors corresponding to the feature points of the downsampled RGB furniture images; using the feature points that meet the set similarity value as the matching feature points of the two downsampled RGB furniture images, and constructing a set of matching feature points based on the matching feature points of every two downsampled RGB furniture images. Based on the set of matching feature points, the two downsampled RGB furniture images with the most matching feature points are selected, and the matching feature points are sorted in descending order based on the cosine similarity value of all matching feature points of the two downsampled RGB furniture images to obtain the matching feature point sequence. The first p matching feature points in the matching feature point sequence are selected to determine the initial matrix of camera extrinsic parameters; An initial point cloud is generated based on the matching feature points of the two downsampled RGB furniture images with the most matching feature points. Using the BA optimization algorithm, the downsampled RGB furniture images with the most matching feature points from the set of matching feature points are selected sequentially. Based on the matching feature points and corresponding feature vectors of these downsampled RGB furniture images, the initial point cloud and the initial camera extrinsic matrix are optimized. This process continues until all downsampled RGB furniture images have been optimized for both the initial point cloud and the initial camera extrinsic matrix. Finally, BA optimization is performed on the optimized initial point cloud and the optimized initial camera extrinsic matrix to obtain the camera extrinsic matrix and the initial sparse point cloud.

4. The method for three-dimensional reconstruction and real-time editing of indoor furniture according to claim 2, characterized in that, Based on the camera intrinsic matrix, the camera extrinsic matrix, and the RGB furniture image, the sparse point cloud is rendered and iteratively trained according to a set number of iterations to obtain a 3D Gaussian model, including: An initial three-dimensional Gaussian model is constructed based on the sparse point cloud; The point cloud in the RGB furniture image corresponding to the camera intrinsic parameter matrix is ​​obtained from the sparse point cloud, and the point cloud in the RGB furniture image is rendered using the volume rendering equation to obtain the rendered initial three-dimensional Gaussian model. Based on the RGB furniture image, the backpropagation algorithm is used to iteratively train the rendered initial 3D Gaussian model according to a set number of iterations until the training reaches the set number of iterations. The trained initial 3D Gaussian model is then used as the 3D Gaussian model.

5. The method for three-dimensional reconstruction and real-time editing of indoor furniture according to claim 1, characterized in that, Based on the set requirements, the 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements, including: Based on the set requirements, the 3D Gaussian model is loaded into the initial scene and rendered and visualized. Based on the set requirements, the rendered and visualized 3D Gaussian model is edited in real time in the initial scene to obtain a home decoration scene that meets the set requirements.

6. The method for three-dimensional reconstruction and real-time editing of indoor furniture according to claim 5, characterized in that, The means of real-time editing of the three-dimensional Gaussian model include: translating, rotating and / or scaling the three-dimensional Gaussian model.

7. A system for three-dimensional reconstruction and real-time editing of indoor furniture, characterized in that, include: The data acquisition module is used to acquire furniture image data from multiple angles in real time; The data preprocessing module is used to obtain the camera intrinsic parameter matrix, camera extrinsic parameter matrix, sparse point cloud, and RGB furniture image based on the furniture image data; The pixel rendering module is used to render the sparse point cloud based on the camera intrinsic parameter matrix, the camera extrinsic parameter matrix and the RGB furniture image, and to perform iterative training according to a set number of iterations to obtain a three-dimensional Gaussian model. The visual editing module is used to perform real-time editing of the three-dimensional Gaussian model in the initial scene based on set requirements, so as to obtain a home decoration scene that meets the set requirements.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method for three-dimensional reconstruction and real-time editing of indoor furniture as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method for three-dimensional reconstruction and real-time editing of indoor furniture as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the method for three-dimensional reconstruction and real-time editing of indoor furniture as described in any one of claims 1-6.

Images