A method and system for real-time three-dimensional reconstruction of tobacco based on Gaussian modeling

Abstract

A method and system for real-time 3D reconstruction of tobacco shreds based on Gaussian modeling is disclosed. The method includes: 1. Constructing an initial tobacco shred point cloud; 2. Assigning Gaussian particle visual features to each point in the initial tobacco shred point cloud to establish an initial 3D Gaussian model; 3. Jointly optimizing the 3D Gaussian model using multiple loss functions; 4. Removing outliers from the tobacco shred point cloud that do not conform to the geometric structure; 5. Introducing noise into the Gaussian particle visual features to generate a rendered image with degradation features, constructing a training set, and training a repair network using the training set; 6. Defining the space where multiple cameras are located in the constructed elliptical trajectory as the reference path, and the space where no cameras are located in the elliptical trajectory as the repair path, and jointly optimizing the rendered images obtained on the repair path and the reference path using the trained repair network to obtain the final tobacco shred point cloud. This invention achieves real-time generation of tobacco shred point clouds, significantly improving the quality and detail of point cloud reconstruction.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction and scanning technology, and in particular to a method and system for real-time three-dimensional reconstruction of tobacco shreds based on Gaussian modeling. Background Technology

[0002] In tobacco production, parameters such as the moisture content, structure, and filler value of tobacco shreds have a significant impact on cigarette quality and production costs. Among these, the filler value is a key factor affecting cigarette appearance, draw resistance, smoking experience, and production costs. It not only reflects the economic efficiency of cigarette production but is also closely related to raw material consumption. Therefore, accurately measuring the volume of tobacco shreds, especially accurately assessing the volume of stored tobacco shreds, is crucial for enterprise production management. Through effective volume measurement data, enterprises can better manage assets, optimize resource allocation, thereby improving modern management practices and reducing waste.

[0003] Traditional methods for measuring tobacco volume typically rely on manual operation, especially for measuring tobacco piles, where traditional methods have many shortcomings. Due to the complex shape and irregular surface of tobacco piles, manual measurement is not only inefficient but also prone to significant measurement errors. Furthermore, manual measurement methods are poorly adaptable to environmental conditions; in complex production environments and with varying pile shapes, the accuracy and repeatability of measurement results often cannot be guaranteed. These problems not only affect production efficiency but may also lead to increased production costs and wasted resources. Because the surface shape of a pile of tobacco is irregular, the first step in measuring its volume is to accurately and effectively obtain three-dimensional data of the pile's surface. Inaccurate three-dimensional data of the tobacco pile surface will result in significant errors in the measured tobacco volume.

[0004] Therefore, there is an urgent need for a method to acquire three-dimensional data of the surface of tobacco piles in real time. Summary of the Invention

[0005] This invention provides a method and system for real-time three-dimensional reconstruction of tobacco shreds based on Gaussian modeling, in order to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] This invention provides a real-time 3D reconstruction method for tobacco shreds based on Gaussian modeling, comprising the following steps:

[0008] S1. Use multiple cameras to acquire multi-view images of tobacco shreds, obtain multi-view tobacco shred images, and construct an initial tobacco shred point cloud based on the multi-view tobacco shred images;

[0009] S2. Assign Gaussian particle visual features, including position, color, Gaussian scale, rotation and transparency, to each point in the initial tobacco point cloud, and establish an initial 3D Gaussian model based on the initial tobacco point cloud and the Gaussian particle visual features of each point.

[0010] S3. Optimize the initial 3D Gaussian model by jointly using multiple loss functions;

[0011] S4. Use the KNN algorithm to remove outliers in the current tobacco point cloud that do not conform to the geometric structure.

[0012] S5. Introduce 3D noise into the Gaussian particle visual features of each point in the current tobacco point cloud to generate a rendered image with degradation features, construct a training set, and use the training set to train the repair network.

[0013] S6. Construct a virtual elliptical trajectory around the distribution of tobacco shreds. Define the space where multiple cameras are located in the elliptical trajectory as the reference path and the space where cameras are not located in the elliptical trajectory as the repair path. Use the trained repair network to jointly optimize the rendered images obtained on the repair path and the reference path to obtain the final tobacco shred point cloud.

[0014] Furthermore, step S1 specifically includes the following steps:

[0015] S11. Collect multi-view images of tobacco shreds from multiple cameras at different angles to obtain tobacco shred image data;

[0016] S12. Then, the visual segmentation model SAM is used to segment the target region of the tobacco image in the tobacco image data to obtain the tobacco mask image under each viewpoint.

[0017] S13. Based on the camera's intrinsic and extrinsic parameters, back-project each tobacco mask image into three-dimensional space, construct the viewing cone region corresponding to each viewpoint, and take the intersection of the four viewing cones as the possible spatial range where the tobacco may exist to obtain the visual shell.

[0018] S14. Uniform sampling within the visual shell Let there be three-dimensional points, denoted as... ,in R represents the i-th three-dimensional sampling point; R represents the set of real numbers;

[0019] S15. Transfer each 3D sampling point Each 3D sampling point is back-projected back onto the 2D plane of its respective viewpoint to determine its position. The projection points are determined by whether they all lie within the mask area of ​​the tobacco mask image corresponding to each viewpoint. If the projection points lie within the mask area in all views, they are retained; otherwise, they are discarded. This yields the initial tobacco point cloud. .

[0020] Furthermore, step S2 specifically includes the following steps:

[0021] S21. Add the initial tobacco to the cloud. Each point is back-projected into the tobacco image at each viewpoint. Bilinear interpolation is used to obtain the color value corresponding to each viewpoint at the current point. Spherical harmonic expansion is then used to encode the color value of each viewpoint into a set of spherical harmonic color coefficients. ;

[0022] S22. Calculate the initial tobacco point cloud. The average Euclidean distance between each point and its K nearest neighbors is used as an estimate of the spatial scale of that point. This estimate of the spatial scale of each point is the Gaussian scale of that point. ;

[0023] S23, Initial tobacco cloud The rotation parameters of each point are initialized using a unit quaternion [1, 0, 0, 0], meaning each point is initially in a state without rotation, resulting in the quaternion rotation of each point. ;

[0024] S24. Set the initial tobacco dot cloud. The transparency of each point is obtained. ;

[0025] S25, Based on the initial tobacco cloud pattern and the initial tobacco cloud Spherical harmonic color coefficients at each point Gaussian scale Quaternion rotation ,transparency Construct the initial 3D Gaussian model.

[0026] Furthermore, the specific expression for the 3D Gaussian model is as follows:

[0027] ;

[0028] in, This represents a 3D Gaussian model.

[0029] Furthermore, step S3 specifically includes the following steps:

[0030] S31. Render the current 3D Gaussian model from different viewpoints to obtain rendered images, and then calculate the color loss between the rendered images and the real images. The mask loss between the segmented mask region and the real mask region. The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network. ;

[0031] S32. Utilize the color loss between the rendered image and the real image. The mask loss between the segmented mask region and the real mask region. Depth loss between rendered image and predicted depth map Construct the total loss function ;

[0032] S33, Based on the total loss function Adjust the Gaussian particle visual features of each point in the current tobacco point cloud;

[0033] S34, repeat S31 to S33 until the total loss function is reached. Minimize to obtain the optimized 3D Gaussian model.

[0034] Furthermore, the color loss between the rendered image and the real image in S31 The formula for calculation is:

[0035] ;

[0036] in, This represents the rendered color of the j-th pixel. The color of the corresponding pixel in the real image, where M is the total number of pixels;

[0037] Mask loss between the segmented mask region and the real mask region The formula for calculation is:

[0038] ;

[0039] in, For real mask labels, The predicted mask probability for the corresponding pixel in the rendered image;

[0040] The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network. The formula for calculation is:

[0041] ;

[0042] in, This represents the depth value of the rendered image obtained from the current 3D Gaussian model. This indicates the depth estimated by the pre-trained depth estimation network;

[0043] The total loss function The formula for calculation is:

[0044] ;

[0045] in, , and This is a hyperparameter used to control the degree of impact of different losses on the overall optimization.

[0046] Furthermore, step S4 specifically includes the following steps:

[0047] S41, Targeting tobacco cloud formation For each point in the graph, the KNN algorithm is used to find each point and its surrounding neighbors, and the Euclidean distance between them is calculated to obtain the tobacco point cloud. The Euclidean distance between each point in the array;

[0048] S42, Based on the cloud formation of tobacco shreds Calculating the Euclidean distance between each point in the tobacco point cloud The mean distance and standard deviation between each point in the data;

[0049] S43. Set a threshold based on the average distance and standard deviation. The specific formula for calculating the threshold is as follows:

[0050] ;

[0051] in, This indicates the set threshold. Indicates tobacco shreds dotting the clouds The average distance between all points in the middle; Indicates tobacco shreds dotting the clouds The standard deviation between each point in the data; A constant set by humans;

[0052] S44. Identifying Tobacco Clouds Current point in With the current point Surrounding neighboring points Is the Euclidean distance between them greater than a threshold? If so, then set the current point... From tobacco to clouds Remove from the middle;

[0053] S45, repeat S44 until the tobacco is lit. Points whose Euclidean distance between them is greater than a threshold are all removed.

[0054] Furthermore, step S5 specifically includes the following steps:

[0055] S51. The tobacco images obtained in S1 are divided into multiple subsets using the "leave one out" method, and an optimized 3D Gaussian model is trained based on each subset so that the rendered images generated by the trained 3D Gaussian model are closer to the real images. In each round of training, the visual features of Gaussian particles in the 3D Gaussian model are dynamically adjusted, and the rendered images are generated using the current 3D Gaussian model in each round of training.

[0056] S52. Introduce statistically distributed 3D noise into the Gaussian particle visual features within the trained 3D Gaussian model. The 3D noise includes spatial perturbation, color perturbation, and point transparency perturbation; generate a rendered image with degradation features.

[0057] S53. Pair the degraded image with the corresponding rendered image generated in S51 to construct a training set. Then, use the training set to train the repair network to obtain the trained repair network.

[0058] Furthermore, step S6 specifically includes the following steps:

[0059] S61. Connect the spaces where multiple cameras are located to form a virtual elliptical trajectory around the distribution of tobacco shreds. Define the space where multiple cameras are located in the elliptical trajectory as the reference path and the space where cameras are not located in the elliptical trajectory as the repair path.

[0060] S62. Sample tobacco images from a new perspective from the repair path, and then input the tobacco images from the new perspective into the trained 3D Gaussian model to obtain the rendered image from the new perspective.

[0061] S63. Use the trained repair network to repair the rendered image from the new perspective and generate the repaired rendered image along the repair path.

[0062] S64. Introduce a weighting mechanism based on view distance, and construct a weighted total loss based on the repaired rendered image on the repair path, the rendered image on the reference path generated by the current 3D Gaussian model, and the rendered image of the corresponding new viewpoint. Then, dynamically adjust the tobacco point cloud based on the weighted total loss. The visual features of Gaussian particles at each point are used to jointly optimize the rendered images obtained on the repair path and the reference path, as well as to optimize the 3D Gaussian model, to obtain the final tobacco point cloud. .

[0063] In another aspect, the present invention provides a real-time 3D reconstruction system for tobacco shreds based on Gaussian modeling, including a device with multiple cameras, which uses a real-time 3D reconstruction method for tobacco shreds to acquire point clouds of tobacco shreds in real time.

[0064] The beneficial effects of this invention are:

[0065] This invention discloses a real-time 3D reconstruction method for tobacco shreds based on Gaussian modeling. This method effectively handles the irregularities on the surface of tobacco shred piles and obtains the volume of the pile. Compared to traditional manual measurement methods, it not only improves measurement accuracy but also significantly reduces errors introduced by traditional manual measurement. Furthermore, it enables real-time generation of tobacco shred point clouds, significantly improving the quality and detail of point cloud reconstruction and greatly increasing production efficiency. Attached Figure Description

[0066] Figure 1 This is a flowchart of the real-time three-dimensional reconstruction method for tobacco shreds in this invention;

[0067] Figure 2 This is a flowchart of S1 in this invention;

[0068] Figure 3 This is a flowchart of S3 in this invention;

[0069] Figure 4 This is a flowchart of S4 in this invention;

[0070] Figure 5 This is a flowchart of S6 in the present invention;

[0071] Figure 6 This is an example diagram of the reference path and repair path in S6 of this embodiment of the invention. Detailed Implementation

[0072] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0073] Reference Figure 1 This application provides a method for real-time 3D reconstruction of tobacco shreds based on Gaussian modeling, including the following steps:

[0074] S1. Multiple cameras are used to acquire multi-view images of tobacco shreds from different perspectives to obtain multi-view tobacco shred images, and an initial tobacco shred point cloud is constructed based on the multi-view tobacco shred images; the cameras used are binocular cameras.

[0075] S2. Assign Gaussian particle visual features, including position, color, Gaussian scale, rotation and transparency, to each point in the initial tobacco point cloud, and establish an initial 3D Gaussian model based on the initial tobacco point cloud and the Gaussian particle visual features of each point.

[0076] S3. Optimize the initial 3D Gaussian model by jointly using multiple loss functions;

[0077] S4. Use the KNN (K-Nearest Neighbor) algorithm to remove outliers in the current tobacco point cloud that do not conform to the geometric structure;

[0078] S5. Introduce 3D noise into the Gaussian particle visual features of each point in the current tobacco point cloud to generate a rendered image with degradation features, construct a training set, and use the training set to train the repair network.

[0079] S6. Construct a virtual elliptical trajectory around the distribution of tobacco shreds. Define the space where multiple cameras are located in the elliptical trajectory as the reference path and the space where cameras are not located in the elliptical trajectory as the repair path. Use the trained repair network to jointly optimize the rendered images obtained on the repair path and the reference path to obtain the final tobacco shred point cloud.

[0080] This invention can effectively handle the irregularities on the surface of tobacco piles and obtain the volume of the tobacco pile. Compared with traditional manual measurement methods, it not only improves measurement accuracy but also significantly reduces the errors caused by traditional manual measurement. Furthermore, it enables real-time generation of tobacco point clouds, significantly improving the quality and detail of point cloud reconstruction and greatly increasing production efficiency.

[0081] In some embodiments, S1 specifically includes the following steps:

[0082] S11. Obtain viewpoint information through the camera's intrinsic and extrinsic parameters. Acquire multi-view images of the tobacco from multiple cameras at different viewpoints to obtain tobacco image data. The preferred number of viewpoints is four. Specifically, images are acquired from four different angles: I1, I2, I3, and I4. Each image provides information about observing the tobacco from different viewpoints. The flowchart is as follows: Figure 2 As shown;

[0083] S12. Then, the visual segmentation model SAM (Segment Anything Model) is used to segment the target region of the tobacco image in the tobacco image data to obtain the tobacco mask image under each view.

[0084] Specifically, the visual segmentation model SAM achieves high-precision segmentation of tobacco images by fusing image encoding and cue information and combining a mask decoding mechanism, obtaining tobacco mask images from each viewpoint. The cue information here is the tobacco point information.

[0085] S13. Based on the camera's intrinsic and extrinsic parameters, back-project each tobacco mask image into three-dimensional space, construct the viewing cone region corresponding to each viewpoint, and take the intersection of the four viewing cones as the possible spatial range where the tobacco may exist to obtain the visual shell.

[0086] S14. Uniform sampling within the visual shell Let there be three-dimensional points, denoted as... ,in R represents the i-th three-dimensional sampling point; R represents the set of real numbers;

[0087] S15. Transfer each 3D sampling point Each 3D sampling point is back-projected back onto the 2D plane of its respective viewpoint to determine its position. The projection points are determined by whether they all lie within the mask area of ​​the tobacco mask image corresponding to each viewpoint. If the projection points lie within the mask area in all views, they are retained; otherwise, they are discarded. This yields the initial tobacco point cloud. This process is used for subsequent modeling and Gaussian parameter initialization. It ensures that the tobacco point cloud only covers the tobacco outline area, effectively removing background interference and improving the accuracy of 3D modeling. (Tobacco point cloud) The specific calculation formula is as follows:

[0088] ;

[0089] in, ) indicates that three-dimensional points Projected to the An image plane from a single perspective; It is the first The tobacco mask region from each perspective, that is, the area where the tobacco is located in the tobacco mask image.

[0090] In some embodiments, S2 specifically includes the following steps:

[0091] S21. Add the initial tobacco to the cloud. Each point in the image is back-projected into the tobacco image from each viewpoint. Here, back-projection refers to using the camera's extrinsic parameters (rotation matrix and translation vector) to back-project the tobacco point cloud. The points (3D points) in the image are transformed from the world coordinate system to the camera coordinate system, and then projected onto the 2D image plane through the camera's intrinsic parameters (intrinsic parameter matrix), thus obtaining the tobacco point cloud. The corresponding pixel position of the point in each viewpoint image;

[0092] Based on tobacco cloud Each point corresponds to a pixel position, and bilinear interpolation is used to obtain the color value corresponding to each viewpoint of the current point. Then, spherical harmonic expansion is used to encode the color value of each viewpoint into a set of spherical harmonic color coefficients. It is used to characterize the distribution of color in different directions;

[0093] S22. Calculate the initial tobacco point cloud. The average Euclidean distance between each point and its K nearest neighbors is used as an estimate of the spatial scale of that point. This estimate of the spatial scale of each point is the Gaussian scale of that point. Gaussian scale Used to control its spatial coverage; Gaussian scale It can reflect the extent and ambiguity of the current point's influence in space;

[0094] S23, Initial tobacco cloud The rotation parameters of each point are initialized using a unit quaternion [1, 0, 0, 0], meaning each point is initially in a state without rotation, resulting in the quaternion rotation of each point. Quaternion rotations can provide a neutral starting point for subsequent learning and optimization.

[0095] S24. Set the initial tobacco dot cloud. The transparency of each point is obtained. Preferably, the initial transparency is set to 0.05. ,in, Indicates the initial transparency, used to control its visibility during the rendering process;

[0096] S25, Based on the initial tobacco cloud pattern and the initial tobacco cloud Spherical harmonic color coefficients at each point Gaussian scale Quaternion rotation ,transparency Construct the initial 3D Gaussian model.

[0097] In some embodiments, the expression for the 3D Gaussian model is as follows:

[0098] ;

[0099] in, This represents a 3D Gaussian model. The initial 3D Gaussian model provides a structurally complete parameter initialization for subsequent rendering consistency optimization, enabling the tobacco point cloud to not only have a geometric distribution but also incorporate the visual features of Gaussian particles at each point in the tobacco point cloud.

[0100] In some embodiments, refer to Figure 3 S3 specifically includes the following steps:

[0101] S31. Render the current 3D Gaussian model from different viewpoints to obtain rendered images, and then calculate the color loss between the rendered images and the real images. The mask loss between the segmented mask region and the real mask region. The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network. ;

[0102] S32. Utilize the color loss between the rendered image and the real image. The mask loss between the segmented mask region and the real mask region. The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network (ZoeDepth). Construct the total loss function ;

[0103] The process of obtaining the predicted depth map is as follows: First, the tobacco image is input into a pre-trained depth estimation network, and then the distance from each point in the scene to the camera is estimated to obtain the predicted depth map.

[0104] S33, Based on the total loss function Adjust the Gaussian particle visual features of each point in the current tobacco point cloud;

[0105] S34, repeat S31 to S33 until the total loss function is reached. The minimum Gaussian particle visual features of each point in the current tobacco point cloud are continuously optimized through gradient backpropagation, gradually converging to a 3D model that can faithfully reproduce the appearance and structure of the tobacco (i.e., the optimized 3D Gaussian model). This process significantly improves multi-view... Figure 1 Consistency and structural realism.

[0106] In some embodiments, the color loss between the rendered image and the real image in S31 The formula for calculation is:

[0107] ;

[0108] in, This represents the rendered color of the j-th pixel. The color of the corresponding pixel in the real image, where M is the total number of pixels;

[0109] Mask loss between the segmented mask region and the real mask region The formula for calculation is:

[0110] ;

[0111] in, For real mask labels, The predicted mask probability for the corresponding pixel in the rendered image; the mask loss between the segmented mask region and the real mask region. Used to improve the consistency between the reconstructed outline and the actual tobacco shred boundaries;

[0112] Deep loss Monocular depth estimation is introduced as supervision. Monocular depth estimation refers to estimating the distance from each point in the scene to the camera using a pre-trained depth estimation network (ZoeDepth). L1 loss is used to measure the error between the rendered image and the predicted depth image. The formula for calculation is:

[0113] ;

[0114] in, This represents the depth value of the rendered image obtained from the current 3D Gaussian model. This indicates the depth estimated by the pre-trained depth estimation network;

[0115] The total loss function The formula for calculation is:

[0116] ;

[0117] in, , and This is a hyperparameter used to control the degree of impact of different losses on the overall optimization.

[0118] In some embodiments, refer to Figure 4 S4 specifically includes the following steps:

[0119] S41, Targeting tobacco cloud formation For each point in the graph, the KNN algorithm is used to find each point and its surrounding neighbors, and the Euclidean distance between them is calculated to obtain the tobacco point cloud. The Euclidean distance between each point in the matrix and its nearest neighbors is calculated using the following formula:

[0120] ;

[0121] in, , , and , , They represent tobacco shreds and cloud patterns respectively. Points in and points 3D coordinates; Indicates tobacco shreds dotting the clouds Points in and points Euclidean distance;

[0122] S42, Based on the cloud formation of tobacco shreds Calculating the Euclidean distance between each point in the tobacco point cloud The mean distance and standard deviation between each point in the matrix; where the mean distance is calculated as follows:

[0123] ;

[0124] Where N is the number of Gaussian points and K is the number of nearest neighbors; Indicates tobacco shreds dotting the clouds The average distance between each point in the array;

[0125] The tobacco cloud The formula for calculating the standard deviation between each point in the matrix is:

[0126] ;

[0127] in, Indicates tobacco shreds dotting the clouds The standard deviation between each point in the data;

[0128] S43. Set a threshold based on the average distance and standard deviation. The specific formula for calculating the threshold is as follows:

[0129] ;

[0130] in, This indicates the set threshold. Indicates tobacco shreds dotting the clouds The average distance between all points in the middle; Indicates tobacco shreds dotting the clouds The standard deviation between each point in the data; A constant set by humans to represent the acceptable range of fluctuations. It is usually set to 2 or 3;

[0131] S44. Identifying Tobacco Clouds Current point in With the current point Surrounding neighboring points Is the Euclidean distance between them greater than a threshold? If so, then set the current point... From tobacco to clouds Remove from the middle;

[0132] S45, repeat S44 until the tobacco is lit. Points with an Euclidean distance greater than a threshold are all removed; this process removes noise points that do not belong to the tobacco shreds, thus improving the tobacco shred point cloud. The quality and geometric accuracy.

[0133] In some embodiments, S5 specifically includes the following steps:

[0134] S51. The tobacco images obtained in S1 are divided into multiple subsets using the "leave one out" method, and an optimized 3D Gaussian model is trained based on each subset so that the rendered images generated by the trained 3D Gaussian model are closer to the real images. In each round of training, the visual features of Gaussian particles in the 3D Gaussian model are dynamically adjusted, and the rendered images are generated using the current 3D Gaussian model in each round of training.

[0135] S52. Introduce statistically distributed 3D noise into the visual features of Gaussian particles in the trained 3D Gaussian model. The 3D noise includes spatial perturbation, color perturbation, and point transparency perturbation; generate a rendered image with degradation features such as blurring, missing, and misalignment.

[0136] S53. Pair the degraded image with the corresponding rendered image generated in S51 (i.e., the rendered image before degradation with the same input as the degraded image). Use the pairing results to build a high-quality training set. The training set is used to guide the restoration network to learn the ability to recover image details from the degraded state and to provide sufficient data support for subsequent high-precision restoration.

[0137] The repair network is then trained using the training set to obtain the trained repair network. The repair network is preferably a Gaussian repair network based on a diffusion model.

[0138] In some embodiments, such as Figure 5 and Figure 6 As shown, step S6 specifically includes the following steps:

[0139] S61. Connect the spaces where multiple cameras are located to form a virtual elliptical trajectory around the distribution of tobacco shreds. Define the spaces where multiple cameras are located within the elliptical trajectory as the reference path, and define the spaces where no cameras are located within the elliptical trajectory as the repair path. See the example diagrams for the reference path and the repair path. Figure 6 As shown, the pink arc represents the repair path, the blue arc represents the reference path, and cameras one through four are on four different reference paths.

[0140] S62. Sample tobacco images from a new perspective from the repair path, and then input the tobacco images from the new perspective into the trained 3D Gaussian model to obtain the rendered image from the new perspective.

[0141] S63. Since these new perspective rendering images were not included in the initial model training, they may contain missing details. Therefore, it is necessary to use the trained inpainting network to inpaint the new perspective rendering images, generating clearer and more complete rendering images along the inpainting path to enhance the continuity and texture consistency between perspectives.

[0142] S64. Introduce a weighting mechanism based on view distance. Specifically, the weighting mechanism based on view distance refers to assigning different importance weights to different repair paths by calculating the spatial distance between each repair path and the reference path view. The closer to the reference path, the greater the weight, because its results are more reliable.

[0143] A weighted total loss is constructed based on the repaired rendered image along the repair path, the rendered image along the reference path generated by the current 3D Gaussian model, and the rendered image from the corresponding new perspective. Then, the tobacco point cloud is dynamically adjusted based on the weighted total loss. The visual features of Gaussian particles at each point are used to jointly optimize the rendered images obtained on the repair path and the reference path, as well as to optimize the 3D Gaussian model, further improving the detail and consistency of the tobacco point cloud; resulting in a refined and consistent final tobacco point cloud. This stage utilizes a distance-aware repair optimization based on weighted total loss, ensuring that the final generated tobacco point cloud has higher accuracy and realism.

[0144] The weighted total loss is as follows:

[0145] ;

[0146] in, denoted as the weighted total loss; w(t) represents the weight function, and the weight function w(t) satisfies w(0)≈1 in the early stage. By controlling the weight function w(t), the focus of the iteration can be shifted, that is, in the early stage of the iteration, the focus is on optimizing the rendering image of the repair path, and in the later stage of the iteration, the focus gradually shifts to the reference path image. This indicates the loss of the repair path from different perspectives; This represents the total loss along the reference path, i.e. ;

[0147] The loss The formula for calculation is:

[0148] ;

[0149] in, This represents the loss of the repair path at the i-th viewpoint; This represents the distance weight; and its calculation formula is:

[0150] ;

[0151] in, This represents the distance between the i-th repaired viewpoint and the nearest viewpoint on the reference path; This represents an exponential function with the natural constant e as its base.

[0152] distance The formula for calculation is:

[0153] ;

[0154] in, Indicates the viewpoint position of the reference path; It represents the 2-norm.

[0155] This invention combines a four-view binocular camera and 3D Gaussian modeling technology to solve the problem of real-time reconstruction of tobacco point clouds. Specifically, by acquiring images from four perspectives and segmenting the target region, an accurate visual shell is constructed, and point cloud sampling and filtering are performed within this space to ensure the effectiveness and geometric accuracy of the tobacco point cloud. Subsequently, 3D Gaussian modeling technology is used for initial point cloud transformation, and the accuracy and consistency of reconstruction are improved by jointly optimizing multiple loss functions. Furthermore, the 3D model is refined by removing outliers. Finally, by combining the degraded image generated from the training set with the restoration of the image using a Gaussian inpainting network based on a diffusion model, real-time generation of tobacco point clouds is successfully achieved, significantly improving the quality and detail of the reconstructed tobacco point cloud.

[0156] In another aspect, the present invention provides a real-time 3D reconstruction system for tobacco shreds based on Gaussian modeling, including a device with multiple cameras, which uses a real-time 3D reconstruction method for tobacco shreds to acquire point clouds of tobacco shreds in real time.

[0157] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for real-time 3D reconstruction of tobacco shreds based on Gaussian modeling, characterized in that, Includes the following steps: S1. Use multiple cameras to acquire multi-view images of tobacco shreds, obtain multi-view tobacco shred images, and construct an initial tobacco shred point cloud based on the multi-view tobacco shred images; S2. Assign Gaussian particle visual features, including position, color, Gaussian scale, rotation and transparency, to each point in the initial tobacco point cloud, and establish an initial 3D Gaussian model based on the initial tobacco point cloud and the Gaussian particle visual features of each point. S3. Optimize the initial 3D Gaussian model by jointly using multiple loss functions; S4. Use the KNN algorithm to remove outliers in the current tobacco point cloud that do not conform to the geometric structure. S5. Introduce 3D noise into the Gaussian particle visual features of each point in the current tobacco point cloud to generate a rendered image with degradation features, construct a training set, and use the training set to train the repair network. S6. Construct a virtual elliptical trajectory around the distribution of tobacco shreds. Define the space where multiple cameras are located in the elliptical trajectory as the reference path and the space where cameras are not located in the elliptical trajectory as the repair path. Use the trained repair network to jointly optimize the rendered images obtained on the repair path and the reference path to obtain the final tobacco shred point cloud.

2. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 1, characterized in that, S1 specifically includes the following steps: S11. Collect multi-view images of tobacco shreds from multiple cameras at different angles to obtain tobacco shred image data; S12. Then, the visual segmentation model SAM is used to segment the target region of the tobacco image in the tobacco image data to obtain the tobacco mask image under each viewpoint. S13. Based on the camera's intrinsic and extrinsic parameters, back-project each tobacco mask image into three-dimensional space, construct the viewing cone region corresponding to each viewpoint, and take the intersection of the four viewing cones as the possible spatial range where the tobacco may exist to obtain the visual shell. S14. Uniform sampling within the visual shell Let there be three-dimensional points, denoted as... ,in R represents the i-th three-dimensional sampling point; R represents the set of real numbers; S15. Transfer each 3D sampling point Each 3D sampling point is back-projected back onto the 2D plane of its respective viewpoint to determine its position. The projection points are determined by whether they all lie within the mask area of ​​the tobacco mask image corresponding to each viewpoint. If the projection points lie within the mask area in all views, they are retained; otherwise, they are discarded. This yields the initial tobacco point cloud. .

3. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 1, characterized in that, S2 specifically includes the following steps: S21. Add the initial tobacco to the cloud. Each point is back-projected into the tobacco image at each viewpoint. Bilinear interpolation is used to obtain the color value corresponding to each viewpoint at the current point. Spherical harmonic expansion is then used to encode the color value of each viewpoint into a set of spherical harmonic color coefficients. ; S22. Calculate the initial tobacco point cloud. The average Euclidean distance between each point and its K nearest neighbors is used as an estimate of the spatial scale of that point. This estimate of the spatial scale of each point is the Gaussian scale of that point. ; S23, Initial tobacco cloud The rotation parameters of each point are initialized using a unit quaternion [1, 0, 0, 0], meaning each point is initially in a state without rotation, resulting in the quaternion rotation of each point. ; S24. Set the initial tobacco dot cloud. The transparency of each point is obtained. ; S25, Based on the initial tobacco cloud pattern and the initial tobacco cloud Spherical harmonic color coefficients at each point Gaussian scale Quaternion rotation ,transparency Construct the initial 3D Gaussian model.

4. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 3, characterized in that, The specific expression for the 3D Gaussian model is as follows: ； in, This represents a 3D Gaussian model.

5. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 4, characterized in that, S3 specifically includes the following steps: S31. Render the current 3D Gaussian model from different viewpoints to obtain rendered images, and then calculate the color loss between the rendered images and the real images. The mask loss between the segmented mask region and the real mask region. The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network. ; S32. Utilize the color loss between the rendered image and the real image. The mask loss between the segmented mask region and the real mask region. Depth loss between rendered image and predicted depth map Construct the total loss function ; S33, Based on the total loss function Adjust the Gaussian particle visual features of each point in the current tobacco point cloud; S34, repeat S31 to S33 until the total loss function is reached. Minimize to obtain the optimized 3D Gaussian model.

6. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 5, characterized in that, The color loss between the rendered image and the real image in S31 The formula for calculation is: ； in, This represents the rendered color of the j-th pixel. The color of the corresponding pixel in the real image, where M is the total number of pixels; Mask loss between the segmented mask region and the real mask region The formula for calculation is: ； in, For real mask labels, The predicted mask probability for the corresponding pixel in the rendered image; The depth loss between the rendered image and the predicted depth map obtained through a pre-trained depth estimation network. The formula for calculation is: ； in, This represents the depth value of the rendered image obtained from the current 3D Gaussian model. This indicates the depth estimated by the pre-trained depth estimation network; The total loss function The formula for calculation is: ； in, , and This is a hyperparameter used to control the degree of impact of different losses on the overall optimization.

7. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 6, characterized in that, S4 specifically includes the following steps: S41, Targeting tobacco cloud formation For each point in the graph, the KNN algorithm is used to find each point and its surrounding neighbors, and the Euclidean distance between them is calculated to obtain the tobacco point cloud. The Euclidean distance between each point in the array; S42, Based on the cloud formation of tobacco shreds Calculating the Euclidean distance between each point in the tobacco point cloud The mean distance and standard deviation between each point in the data; S43. Set a threshold based on the average distance and standard deviation. The specific formula for calculating the threshold is as follows: ； in, This indicates the set threshold. Indicates tobacco shreds dotting the clouds The average distance between all points in the middle; Indicates tobacco shreds dotting the clouds The standard deviation between each point in the data; A constant set by humans; S44. Identifying Tobacco Clouds Current point in With the current point Surrounding neighboring points Is the Euclidean distance between them greater than a threshold? If so, then set the current point... From tobacco to clouds Remove from the middle; S45, repeat S44 until the tobacco is lit. Points whose Euclidean distance between them is greater than a threshold are all removed.

8. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 7, characterized in that, S5 specifically includes the following steps: S51. The tobacco images obtained in S1 are divided into multiple subsets using the "leave one out" method, and an optimized 3D Gaussian model is trained based on each subset so that the rendered images generated by the trained 3D Gaussian model are closer to the real images. In each training round, the visual features of Gaussian particles in the 3D Gaussian model are dynamically adjusted, and in each training round, the current 3D Gaussian model is used to generate rendered images. S52. Introduce statistically distributed 3D noise into the Gaussian particle visual features within the trained 3D Gaussian model. The 3D noise includes spatial perturbation, color perturbation, and point transparency perturbation; generate a rendered image with degradation features. S53. Pair the degraded image with the corresponding rendered image generated in S51 to construct a training set. Then, use the training set to train the repair network to obtain the trained repair network.

9. The real-time three-dimensional reconstruction method for tobacco shreds based on Gaussian modeling according to claim 7, characterized in that, S6 specifically includes the following steps: S61. Connect the spaces where multiple cameras are located to form a virtual elliptical trajectory around the distribution of tobacco shreds. Define the space where multiple cameras are located in the elliptical trajectory as the reference path and the space where cameras are not located in the elliptical trajectory as the repair path. S62. Sample tobacco images from a new perspective from the repair path, and then input the tobacco images from the new perspective into the trained 3D Gaussian model to obtain the rendered image from the new perspective. S63. Use the trained repair network to repair the rendered image from the new perspective and generate the repaired rendered image along the repair path. S64. Introduce a weighting mechanism based on view distance, and construct a weighted total loss based on the repaired rendered image on the repair path, the rendered image on the reference path generated by the current 3D Gaussian model, and the rendered image of the corresponding new viewpoint. Then, dynamically adjust the tobacco point cloud based on the weighted total loss. The visual features of Gaussian particles at each point are used to jointly optimize the rendered images obtained on the repair path and the reference path, as well as to optimize the 3D Gaussian model, to obtain the final tobacco point cloud. .

10. A real-time 3D reconstruction system for tobacco shreds based on Gaussian modeling, characterized in that, The device includes multiple cameras, and the device uses the real-time three-dimensional reconstruction method of tobacco shreds as described in any one of claims 1 to 9 to acquire tobacco shred point clouds in real time.

Images