Method and system for reconstructing three-dimensional digital particles based on single particle scene photo

Abstract

The invention discloses a method and system for reconstructing three-dimensional digital particles based on a single particle scene photograph, and the method comprises the steps: inputting a two-dimensional scene image containing multiple particles into a U-Net auxiliary image segmentation frame, outputting a mask containing all particles in the image, carrying out the cutting and background replacement based on the mask, and carrying out the reconstruction of the three-dimensional digital particles. Standard views of all single particles in the image are extracted; reasoning the standardized views of all the single particles one by one by using a trained multi-view generation model to generate corresponding multi-view images; inputting the multi-view images of all the single particles into the fine-tuned large reconstruction model in sequence to obtain a three-dimensional grid model of all the single particles; and carrying out physical scale reduction on the three-dimensional grid models of all the single particles based on the scaling factors, and then carrying out grid simplification to obtain three-dimensional digital particles corresponding to all the single particles.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of civil and water conservancy engineering, and particularly relates to a method and system for reconstructing three-dimensional digital particles based on a single particle scene photo. BACKGROUND

[0002] The present application focuses on the problem of quickly, automatically and batchly reconstructing the real three-dimensional topography of particles and outputting quantifiable statistical indicators under the condition of providing only a single two-dimensional image containing multiple rock-soil particles. Engineering and scientific applications of particle materials, such as discrete element numerical simulation calibration, material grading and morphology optimization, particle abrasion / crushing mechanism analysis, storage and flow behavior prediction, require the simultaneous acquisition of size distribution, three-dimensional shape parameters and directly usable point cloud or mesh data of a large number of particles.

[0003] However, the particles in a single two-dimensional image usually exhibit high similarity, weak semantics, low texture difference and close packing occlusion. The missing thickness and volume information caused by the covered area and unobserved back surface. The adhesion and missegmentation caused by the adjacent boundaries of multiple instances. At the same time, it is necessary to balance the fidelity of single-particle geometry and the overall high-throughput processing capability while ensuring statistical distribution consistency. Traditional processes that rely on multi-view acquisition or high-cost volume imaging are difficult to meet the above comprehensive requirements in terms of equipment, efficiency and scalability. Strategies based solely on two-dimensional segmentation or general single-view reconstruction often have problems such as geometric flattening, thickness underestimation, occlusion completion distortion and batch statistical deviation, making the obtained data difficult to directly support subsequent detailed simulation and engineering judgment. In view of the above comprehensive technical bottlenecks of incomplete single-view information, high difficulty of instance separation, unreliable occlusion completion and difficult guarantee of statistical consistency, a collaborative reconstruction framework is needed that integrates instance extraction, generates priori and morphological constraints around a single multi-particle image input, realizes high-fidelity, fast and automated reconstruction of a large number of particle three-dimensional topography, and outputs an engineering usable digital particle model. SUMMARY

[0004] In order to overcome the deficiencies of incomplete single-view information, high difficulty of instance separation, unreliable occlusion completion and difficult guarantee of statistical consistency in the prior art of batch reconstruction of real three-dimensional topography of particles based on a single two-dimensional image containing multiple rock-soil particles, the present application provides a method and system for reconstructing three-dimensional digital particles based on a single particle scene photo, which generates a multi-view image set through a multi-view generation model, and realizes the conversion from a single two-dimensional particle image to a high-quality three-dimensional model by combining a large reconstruction model, solves the problems of depth information missing and visual similarity limitation in traditional single-view reconstruction, and provides technical support for digital modeling and physical simulation of particle materials.

[0005] According to an aspect of the present application, a method for reconstructing three-dimensional digital particles based on a single particle scene photo is provided, comprising:

[0006] Input the two-dimensional scene image containing multiple particles into the U-Net assisted image segmentation framework, output a mask containing all particles in the image, perform cropping and background replacement based on the mask, and extract the standardized view of all single particles in the image;

[0007] Use the trained multi-view generation model to infer all single particle standardized views one by one to generate corresponding multi-view images; the multi-view generation model uses the input single particle standardized view as a condition in the inference stage, uses the U-Net architecture to iteratively denoise in the latent space, gradually generates the multi-view image latent representation of the single particle from pure noise, and reconstructs it into a multi-view image through the decoder;

[0008] Input all single particle multi-view images into the fine-tuned large reconstruction model in sequence to obtain three-dimensional mesh models of all single particles;

[0009] Restore the physical scale of all single particle three-dimensional mesh models based on the scaling factor, and then simplify the mesh to obtain three-dimensional digital particles corresponding to all single particles.

[0010] As a further technical solution, the U-Net assisted image segmentation framework processes the input image through the U-Net network to obtain a preliminary mask; the preliminary mask is converted into a point prompt, which is input into the image segmentation base model together with the original input image; in the image segmentation base model, the image embedding of the original input image is extracted through the image encoder; the prompt embedding of the point prompt is extracted through the prompt encoder; the image embedding and the prompt embedding are combined through the mask decoder, and the feature representation is updated using the attention mechanism, and then the output label is mapped to a feature vector through a multi-layer perception, and then the image embedding is operated. Spatial dot product operation generates a mask containing all particles in the image.

[0011] As a further technical solution, the training process of the multi-view generation model includes:

[0012] Use a large-scale three-dimensional data set containing real multi-view images as the training data set;

[0013] Use a diffusion model framework to build a multi-view image generation network;

[0014] The multi-view image generation network is trained using a training data set, and the training process includes: in the forward process, a series of intermediate noise latent representations are generated by gradually adding Gaussian noise to the representation of the real single-particle multi-view image in the latent space in the training data set at each time step until the near pure noise; in the backward inference process, the U-Net architecture is used to iteratively denoise the pure noise at each time step under the condition of inputting the real single-particle standardized view, to generate the representation of a set of single-particle multi-view images corresponding to a pre-defined camera pose in the latent space, and to reconstruct the multi-view image of the single particle through the decoder.

[0015] As a further technical solution, the large reconstruction model adopts an encoder-decoder architecture to learn a three-dimensional representation of the multi-view image of the single particle: the input image is encoded into a patch-level feature sequence by a visual encoder, and the decoder generates an axis-aligned three-plane representation combined with camera parameters; for any three-dimensional sampling point, the feature vector is obtained by projecting it to the three planes for interpolation, and input into a shared multi-layer perceptron to parameterize the neural radiance field to represent the target object; a neural radiance field optimization pipeline with weight inheritance is adopted, and a differentiable isosurface extraction module is integrated to extract a triangular mesh surface from the parameterized neural radiance field to obtain a three-dimensional mesh model of the single particle.

[0016] As a further technical solution, the fine-tuning stage of the large reconstruction model freezes the encoding layer parameters and only updates the decoding layer and three-plane projection related parameters, and uses the multi-view supervision of the fine-tuning training data set, and the fine-tuning training target is composed of pixel-level reconstruction loss, perception loss and mask consistency loss.

[0017] As a further technical solution, the scaling factor is selected according to the aspect ratio and roundness threshold of the particle, and at least one of the following is included:

[0018] 、 、

[0019] In the formula, 、 and all represent the scaling factor; is the projection area of the i-th single particle, is the maximum value of the projection area of all single particles in the image; and are the maximum and minimum projection feature scales of the i-th single particle, and are the maximum values of the corresponding maximum and minimum projection feature scales of all single particles in the image; n represents an empirical index value; is the roundness of the i-th single particle, is the maximum value of the roundness of all single particles in the image.

[0020] As a further technical solution, the mesh simplification process is as follows: the three-dimensional mesh model of a single particle after physical scale restoration is processed by an algorithm based on a quadratic error metric to reduce it to the target number of vertices and faces.

[0021] According to another aspect of this specification, a system for reconstructing three-dimensional digital particles based on a single grain scene photograph is provided, comprising:

[0022] The particle segmentation module is used to input a two-dimensional scene image containing multiple particles into the U-Net-assisted image segmentation framework, output a mask containing all particles in the image, perform cropping and background replacement based on the mask, and extract a normalized view of all single particles in the image.

[0023] The particle multi-view generation module is used to infer the standardized views of all single particles one by one using the trained multi-view generation model to generate the corresponding multi-view images. During the inference phase, the multi-view generation model uses the input standardized single-particle views as conditions and adopts the U-Net architecture to iteratively denoise in the latent space to gradually generate the multi-view image latent representation of the single particle from pure noise, and then reconstructs it into a multi-view image through the decoder.

[0024] The particle 3D reconstruction module is used to sequentially input the multi-view images of all individual particles into the fine-tuned large reconstruction model to obtain the 3D mesh model of all individual particles.

[0025] The particle reconstruction and correction module is used to physically scale all individual particle 3D mesh models based on scaling factors, and then simplify the mesh to obtain the corresponding 3D digital particles for all individual particles.

[0026] According to another aspect of this specification, an electronic device is provided, including a memory and a processor, the memory storing program instructions executed by the processor, the processor invoking the program instructions to perform a method for reconstructing three-dimensional digital particles based on a single grain scene photograph.

[0027] According to another aspect of this specification, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform a method for reconstructing three-dimensional digital particles based on a single grain scene photograph.

[0028] Compared with existing technologies, the beneficial effects of this invention are as follows: Addressing the problem of weak semantics / high similarity among particles, a U-Net-assisted image segmentation framework is proposed. A feature-optimized instance extraction strategy is used to obtain stable multi-particle instance masks and contour shape representations, and images of each particle are saved based on mask cropping. Based on an improved single-view 3D reconstruction backbone, a diffusion-based generation prior and geometric depth compensation mechanism are introduced, proposing a multi-view generation model to generate a multi-view image set. Conditionally constrained morphological inference is performed on unobserved surfaces and occluded areas, thereby alleviating the thickness flattening problem caused by single-view uncertainty. Subsequently, combined with a large-scale reconstruction model, the conversion from a single 2D particle image to a high-quality 3D model is achieved, solving the problems of missing depth information and visual similarity limitations in traditional single-view reconstruction, and providing technical support for the digital modeling and physical simulation of particulate materials.

[0029] This invention proposes a generative AI-assisted 3D reconstruction framework for single-image particle scene reconstruction. The framework revolves around the process of "particle scene image—particle instance segmentation—single-particle multi-view generation—3D reconstruction—batch output," achieving automated 3D reconstruction of massive amounts of particles. Validation on single-image rendered samples from a large-scale real digital particle dataset demonstrates significant improvements in 3D shape reconstruction error, reasonableness of occluded part completion, statistical distribution preservation, and processing throughput, meeting the application needs of rapid engineering analysis and data asset construction. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a method for reconstructing three-dimensional digital particles based on a single particle scene photograph, provided in an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram of the U-Net-assisted SAM segmentation framework in an embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the training process of the multi-view generation model in an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the computational process framework for reconstructing a granular 3D mesh model from a 2D image based on a large reconstruction model in an embodiment of the present invention;

[0035] Figure 5This is a schematic diagram of the technical framework for transforming a single rock and soil particle scene photograph into simulateable three-dimensional digital particles in an embodiment of the present invention;

[0036] Figure 6 This is an example diagram of particle 3D mesh model reconstruction based on a set of two-dimensional particle multi-view images in an embodiment of the present invention;

[0037] Figure 7 This is a schematic diagram of scale reduction and QEM mesh simplification in an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram illustrating discrete element simulation using three-dimensional digital particles in an embodiment of the present invention;

[0039] Figure 9 This is a schematic diagram of the structure of a system for reconstructing three-dimensional digital particles based on a single particle scene photograph, provided in an embodiment of the present invention.

[0040] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0041] It should be noted that:

[0042] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0045] This invention forms an end-to-end reconstruction path for multi-particle sets by coupling instance segmentation, diffusion generation prior, morphological statistical constraints and depth geometric completion mechanism under single-view conditions, realizing a technical path from single particle scene image as input to batch of three-dimensional high-fidelity digital particles as output.

[0046] like Figure 1 As shown, a method for reconstructing 3D digital particles based on a single grain scene photograph includes:

[0047] Step 1: Input the 2D scene image containing multiple particles into the U-Net-assisted image segmentation framework, output a mask containing all particles in the image, perform cropping and background replacement based on the mask, and extract the normalized view of all single particles in the image.

[0048] Step 2: Use the trained multi-view generation model to reason about each of the standardized views of a single particle to generate the corresponding multi-view images.

[0049] Step 3: Input the multi-view images of all individual particles into the fine-tuned large reconstruction model in sequence to obtain the three-dimensional mesh model of all individual particles;

[0050] Step 4: Physically scale all individual particle 3D mesh models based on scaling factors, then simplify the mesh to obtain the corresponding 3D digital particles for all individual particles.

[0051] In step 1, the U-Net-assisted image segmentation framework processes the input image through the U-Net network to obtain a preliminary mask. The preliminary mask is then converted into point cues and input into the SegmentAnything Model (SAM) along with the original input image. In the SAM, the image encoder extracts the image embedding of the original input image, the cue encoder extracts the cue embedding of the point cues, and the mask decoder combines the image embedding and the cue embedding, updates the feature representation using an attention mechanism, and then maps the output label to a feature vector through a multilayer perceptron. Finally, a spatial dot product operation is performed with the image embedding to generate a mask containing all particles in the image.

[0052] Specifically, step 1 is essentially the particle segmentation stage. A two-dimensional scene photo containing multiple particles is acquired, and instance segmentation is performed to obtain a fine binary mask for each particle in the scene photo. For each instance particle, cropping and background displacement are performed based on its mask region to generate a single-particle image, and the single-particle image is saved as a normalized view to obtain a normalized single-particle view.

[0053] The goal of the particle segmentation stage is to extract the shape and texture information of 2D particles from a 2D scene image containing multiple particles captured by a camera. To achieve this, a lightweight U-Net architecture with an encoder-decoder architecture and skip connections is used to coarsely segment the particle image, obtaining a preliminary mask. Point cues derived from the preliminary mask are then input into the Segment Anything Model (SAM) along with the original image. SAM is built on the backbone network of the Vision Transformer (ViT-H) and consists of three main components: an image encoder, a cue encoder, and a lightweight mask decoder. The entire process is as follows: Figure 2 As shown. After segmentation, each particle is extracted and assigned a unique spatial identifier based on its centroid position within the regular grid. Each particle is saved as a separate image file, i.e., a normalized view of a single particle, to maintain the correspondence between the numbering and spatial representation in subsequent 3D reconstruction.

[0054] In step 2, the multi-view generation model is a conditional diffusion model. During the inference stage, the input single-particle standardized view is used as a condition. The U-Net architecture is used to iteratively denoise the latent space, gradually generating single-particle multi-view image latent representations from pure noise, and then reconstructing them into multi-view images by the decoder.

[0055] Optionally, multiple views can be a predefined set of views and / or an adaptively selected set of views.

[0056] Step 2 is essentially the multi-view generation stage for particles. This stage aims to use generative AI, such as the Denoising Diffusion Probabilistic Model (DDPM), to generate images of particles from other perspectives, taking a single-view image of the particle as input. This model employs a conditional diffusion approach. For each standardized view of a single particle obtained in Step 1, it uses this standardized view as a condition to synthesize a set of multi-view images of the particle in the reverse denoising iteration.

[0057] Further, in step 2, the training process of the multi-view generation model includes:

[0058] A large-scale 3D dataset containing real single-particle multi-view images was used as the training dataset;

[0059] Based on generative AI, a multi-view image generation network is constructed using a diffusion model framework;

[0060] The training process of the multi-view image generation network using the training dataset includes: in the forward process, Gaussian noise is progressively added to the representation of real single-particle multi-view images in the latent space of the training dataset at each time step to generate a series of intermediate noise latent representations until near-pure noise; in the reverse process, the pure noise is iteratively denoised using the U-Net architecture with the input real single-particle normalized view as the condition, generating a set of single-particle multi-view image representations in the latent space corresponding to the predefined camera pose, and then reconstructed into single-particle multi-view images by the decoder.

[0061] Specifically, such as Figure 3 As shown. This multi-view generation model is trained through two relative processes:

[0062] Forward Process (Noise Addition): From Real Single-Particle Multi-View Starting with, Gaussian noise is gradually added to the intermediate representation in the latent space. And eventually reach the pure noise state Z (corresponding to the colored noise image on the right side of the schematic diagram).

[0063] Reverse process (reasoning): This is the denoising process, starting with random noise Z, and proceeding according to... (X) For corresponding The denoising intermediate representation. The process iteratively denoises the target single-particle multi-view image sequentially, generating a set of latent representations of the single-particle multi-view images with predefined camera poses. These representations are then reconstructed by a decoder into the target multi-view image. At each step, U-Net uses the conditional latent representations of the normalized single-particle view as conditions, predicts and removes current noise through reference attention, ensuring that the generated multi-view image is geometrically and texturally identical to the input single-view image. Figure 1 Specifically, at each time step, a single-view reference image is injected with noise of the same duration as the current time step, and its latent features are incorporated into the self-attention layer of U-Net via reference attention. This mechanism provides both local texture constraints and global semantic constraints, stabilizing cross-view consistency.

[0064] Through this iterative denoising process, the model gradually synthesizes a set of potential representations of multi-view images corresponding to new viewpoints of a single particle. The single-view images provide crucial conditional information, guiding U-Net to generate multi-view outputs consistent with the particle appearance and structure depicted in the input.

[0065] During training, real single-particle multi-view images from a large-scale 3D dataset are used as supervision signals. Objective functions, including pixel reconstruction loss and perceptual feature loss, are employed to promote cross-view consistency and improve perceptual quality, thereby obtaining a diffusion generation model that can synthesize consistent multi-views under single-view conditions.

[0066] In step 3, the Large Reconstruction Model (LRM) is a dedicated large model for particle reconstruction tasks. It employs an encoder-decoder architecture to learn the 3D representation of multi-view images of a single particle: the visual encoder encodes the input image into a patch-level feature sequence, and the decoder combines camera parameters to generate an axis-aligned three-plane representation; for any 3D sampling point, it is projected onto the three planes and interpolated to obtain a feature vector, which is then input into a shared multilayer perceptron (MLP) to parameterize the neural radiation field (NeRF) to characterize the target object; a weighted NeRF optimization pipeline is used, integrating a differentiable isosurface extraction module (FlexiCubes) to extract triangular mesh surfaces from the parameterized NeRF, resulting in a 3D mesh model of the single particle. Figure 4 The computational workflow framework of the large reconstruction model is shown.

[0067] Specifically, the three planes are three mutually orthogonal and axis-aligned two-dimensional feature planes, and each plane contains learnable position embeddings to enhance the spatial context. The three-plane representation is the corresponding three mutually orthogonal and axis-aligned two-dimensional feature maps.

[0068] Neural radiation fields are used to map spatial coordinates (and viewable directions) to color and voxel density, and are supervised by volume rendering and alignment with real or generated multi-view images. Based on this, a signed distance function (SDF) is introduced to obtain explicit surface representations. The SDF is initialized with the trained NeRF density branch weights and obtained after sign and threshold alignment. It is then combined with a differentiable isosurface extraction method (Flexicubes) to generate explicit meshes, improving geometric usability and downstream visualization / simulation / editing performance. The fine-tuning process employs a strategy of freezing the visual encoding layer and fine-tuning the decoding layer and the three-plane feature projection layer. The supervision targets include pixel reconstruction loss and perceptual feature loss, and may further include mask consistency loss, surface normal consistency loss, and voxel / density sparsity regularization to enhance cross-view consistency and geometric fidelity.

[0069] Preferably, during the fine-tuning and inference process of the large reconstruction model, the extracted mesh is directly rendered in a differentiable manner, and geometric supervision such as depth and normals is applied.

[0070] Step 3 is essentially the fine-tuning of the large-scale reconstruction model and the 3D reconstruction stage. Specifically, the large-scale reconstruction model (LRM) adopts a Transformer-based encoder-decoder architecture, with the visual encoder being a pre-trained visual Transformer (DINO-ViT).

[0071] After the decoder generates a three-plane representation, it projects the points in 3D space to be predicted during volume rendering onto three axis-aligned feature planes. , , The feature vector of the point is obtained through bilinear interpolation; then, the features are decoded by a shared multilayer perceptron (MLP) to output the RGB color of the point. and volume density Through training, the MLP learns an implicit continuous function. It can calculate color and density values ​​in real time based on any 3D coordinates and viewing direction. In three-dimensional coordinates, Direction of sight RGB colors For volume density, finally render the integral using volume rendering. The final pixel color is synthesized along the light rays. For explicit mesh extraction, the continuous implicit density field needs to be pre-sampled on a 128³ regular mesh to form a discretized density field representation, meeting the input requirements of the FlexiCubes isosurface extraction algorithm. Preferably, a weight inheritance mechanism is used during explicit surface extraction and mesh optimization to reuse the original density MLP for SDF prediction. This is achieved through weight transformation. The conversion from a density field to an SDF field is achieved, transforming the object surface in the original density field into a 0-level set of the SDF field, where s represents the weight, d represents the distance, and τ is a pre-defined threshold. FlexiCubes extracts triangular meshes of the surface based on this SDF field at a 128³ mesh resolution. During training, differentiable rasterization rendering is used to render the triangular meshes as 2D images from multiple perspectives. Through multi-supervised comparison with the corresponding real images in terms of RGB, depth, and normals, the consistency constraints of multiple perspectives ensure that the gradients lost during rendering are accurately backpropagated to the vertex coordinates of the 3D mesh. The mesh geometry is adjusted through gradient descent, and then multi-view rendering is performed again, forming an iterative optimization loop of "rendering-comparison-backpropagation-adjustment-re-rendering". This achieves end-to-end optimization of the mesh geometry, enabling the extracted explicit mesh to better reconstruct the complete 3D geometry and appearance of the target object.

[0072] Preferably, to improve the morphology representation and reconstruction accuracy in 3D particle reconstruction scenarios, a high-quality particle fine-tuning training dataset for sand particles is constructed. Optionally, 2,000 samples are selected from a candidate library of approximately 80,000 synthetic particle samples, and processed to obtain high-quality particle data to form a fine-tuning training set, including the following steps:

[0073] Perform hierarchical quality screening on candidate samples, including at least the following criteria: remove samples with missing texture maps, remove samples whose field of view occupies less than 10% in any viewpoint to avoid supervised sparsity and rendering noise introduced by tiny targets, and remove samples with multiple disconnected geometric components to meet single-particle topology constraints.

[0074] For the selected samples, geometric normalization is performed. Optionally, the geometric center is registered to the origin [0,0,0] using the surface mesh vertices as a reference, and the scale is normalized so that the target bounding box is embedded in the world coordinate system [-1,1]. 3 Within this range, the impact of differences between the original pose and scale on training stability and generalization is eliminated.

[0075] To construct multi-view supervision, Blender was used to randomly collect 32 camera poses for each geometrically normalized sample and render them. The camera optical axis pointed to the target and covered a variety of azimuth and pitch angle distributions. The output resolution was 1024×1024, and the camera intrinsic and extrinsic parameters and foreground mask strictly registered with the sample were exported simultaneously as high-quality granular data to provide pixel-level reconstruction and mask consistency supervision signals.

[0076] Furthermore, based on this fine-tuned training set, the general large reconstruction model is fine-tuned for the particle domain, transferring its parameterized representation to the particle morphology distribution, forming a particle-oriented large reconstruction model. During the fine-tuning stage, the encoding layer parameters are frozen, and only the decoding layer and triplane projection-related parameters are updated. High-quality particle data from the fine-tuning training dataset are used for multi-look supervision. The fine-tuning training objective consists of pixel-level reconstruction loss, perceptual loss, and mask consistency loss, with a total loss of [missing information]. for:

[0077] (1)

[0078] in, and They represent the first Predicted images and real images from different perspectives. and These represent the corresponding predicted mask and the ground mask, respectively. and These are the loss weights. During training, the following values ​​can be used: , The initial learning rate is set to 4.0 × 10⁻⁶. −4 Cosine annealing was used to gradually reduce the temperature to 4.0 × 10⁻⁶. −5 By monitoring the consistency metrics between the loss curve and the validation perspective, we can ensure stable model convergence and cross-view consistency.

[0079] Based on a rigorously screened and quality-controlled particle dataset, the Large Reconstruction Model (LRM) is fine-tuned in a domain-specific manner to obtain a dedicated large model for particle reconstruction tasks; the multi-view images are then input into the particle large reconstruction model to reconstruct a three-dimensional mesh model of the particles.

[0080] Step 4 is essentially the particle size correction and mesh simplification stage. Step 3 focuses on shape reconstruction, where LRM normalizes each individual particle to a 1×1×1 cube. The generated 3D mesh model must be restored to its original size using appropriate scaling factors, which are determined by the geometric properties of the original image and the information from the 2D image.

[0081] Specifically, this invention proposes three scaling factors for size correction, which, when restoring the physical scale, include at least one of the following:

[0082] First scaling factor Representing the 3D scaling ratio, defined as:

[0083] (2)

[0084] in, Let be the projected area of ​​the i-th single particle. This represents the maximum projected area of ​​all individual particles in the image. The square root converts the two-dimensional area ratio into a one-dimensional length ratio, which is suitable for particle gradation analysis.

[0085] Second scaling factor Taking into account the aspect ratio, it is defined as:

[0086] ( )

[0087] and These are the maximum and minimum projected feature scales of the i-th single particle, respectively. and , respectively, are the maximum and minimum projected feature scales corresponding to all single particles in the image; n is the empirical index value, which is an empirically adjusted value verified by experiments and must be less than 0.5 to avoid overcorrection of slender or flat particles.

[0088] Third scaling factor Sphericity correction is introduced, defined as:

[0089] ( )

[0090] in, The roundness of the i-th single particle is defined as follows: , It is the perimeter of the particle; It is the maximum roundness of all individual particles in the image.

[0091] The scaling factor is selected based on the particle aspect ratio and roundness threshold. Specifically, when the following conditions are met... and When, priority should be given to using Otherwise adopt or with Weighted combination: in , and This is a preset threshold.

[0092] Furthermore, in step 4, after physically scaling the 3D mesh model of a single particle by calculating the scaling factor based on the projected area, aspect ratio, roundness, and their combination relationships, a mesh simplification process is also included: the physically scaled 3D mesh model of a single particle is processed using an algorithm with a quadratic error metric to reduce it to the target number of vertices and faces. This step is a necessary stage for numerical model conversion to facilitate subsequent analysis and evaluation of particle morphology.

[0093] Specifically, the mesh simplification method employs an edge folding strategy based on the quadratic error metric (QEM): a priority queue is established for candidate edges and they are sorted according to the geometric distortion cost caused by folding. The cost includes a weighted sum of vertex quadratic error, local curvature change, and boundary preservation weights. During iterative folding, topological integrity and boundary / sharp feature constraints are maintained, and cross-boundary folding is prohibited when the local average curvature or feature intensity exceeds a set threshold to preserve sharp edges and shape details. The simplification continues until a predetermined mesh size (target number of vertices and faces) is reached to achieve a uniform number of vertices and faces across different particles for subsequent analysis and evaluation of particle morphology. By sparsening the mesh for each reconstructed 3D digital particle, a balance between geometric accuracy and computational efficiency is ensured.

[0094] During mesh simplification, a quadratic error metric (QEM) algorithm is used to simplify each mesh, reducing it to the target number of vertices and faces. Edge folding is prioritized based on the degree of distortion produced, and this process continues until the mesh reaches the target simplification level. Throughout the simplification process, topological integrity and boundary preservation are maintained to ensure that the simplified mesh remains a true representation of the original structure. All reconstructed particle models are normalized to a uniform number of vertices and faces. This simplification is crucial because it provides a consistent basis for subsequent particle morphology evaluation and facilitates the import of the model into a discrete element model (DEM) for mechanical analysis and numerical simulation.

[0095] Furthermore, exemplarily, a numerical model conversion stage is provided after mesh simplification. This stage focuses on importing the reconstructed 3D particles into a physics-based numerical simulation. Specifically, the scale-corrected mesh (reconstructed 3D digital particles) undergoes mesh simplification using an algorithm based on Quadratic Error Metrics (QEM); an index mapping is established between each 3D particle and its pixel / mask identifier in the original 2D image; and statistical indicators including volume, principal axis length ratio, sphericity, compactness, elongation, flatness, and particle size distribution are calculated and output. In the analysis stage, the statistical distribution of the particle set is compared with the reference distribution using Kullback–Leibler divergence and Jensen–Shannon divergence to quantitatively assess the difference between the generated results and the true distribution, thereby achieving automated batch conversion from a single 2D image to a multi-particle 3D model and statistical results.

[0096] Based on the foregoing technical concept, this invention provides an embodiment: 3D reconstruction of a single mobile phone grainy scene photo into simulateable 3D digital grains, the technical framework of which is as follows: Figure 5 As shown:

[0097] Step (1): Particle Segmentation and Preprocessing. A 2D scene image containing multiple particles is acquired. First, U-Net is used to coarsely segment the particle image, generating cue points and an initial mask. Then, the initial mask is converted into point cues and input into SAM along with the original image, outputting a fine binary mask for each particle. After segmentation, each individual particle is extracted and assigned a unique spatial identifier based on its centroid position within a regular grid. Each individual particle is saved as a separate image file.

[0098] Step (2): Multi-view image generation. For each segmented single-particle view, a conditional diffusion model is used to generate a set of multi-view images of that single particle under a predefined or adaptive viewpoint, based on the single view as a condition.

[0099] Step (3): Construct a large-particle reconstruction model.

[0100] Construct a high-quality training dataset for particles, including screening samples from a large-scale synthetic particle sample library, removing low-quality particles such as those with missing textures or extremely small field of view targets, and performing geometric normalization to ensure data quality. Render the screened samples using physics engines such as Blender, simultaneously outputting camera parameters and foreground masks. Use the above dataset to fine-tune the general large reconstruction model (LRM), freezing the encoding layer and updating only the decoding layer and triplane projection-related parameters. The training objectives include pixel reconstruction loss, perceptual loss, and mask consistency loss. Input the multi-view images generated in step (2) into the fine-tuned LRM to reconstruct the corresponding particle's 3D model, and obtain an explicit triangular mesh through a differentiable isosurface extraction algorithm to obtain a single particle's 3D mesh model, such as... Figure 6 The image shown is an example of reconstructing a 3D mesh model of particles based on a 2D particle set image.

[0101] Step (4): Size Correction. To focus on shape reconstruction, LRM normalizes each particle to a 1×1×1 cube. The generated particles must be restored to their original size using appropriate scaling factors, which are determined by the particle geometry and 2D image information. After 3D reconstruction of the segmented particles, size correction is performed to ensure that the relative size differences between the reconstructed particles better represent the actual situation in the original image.

[0102] Step (5): Mesh simplification. Each mesh is simplified using a Quadratic Error Metric (QEM) algorithm, reducing it to 600 vertices and 1200 faces, as shown below. Figure 7As shown. Edge folding is prioritized based on the degree of distortion produced, and this process continues until the mesh reaches the target simplification level. Throughout the simplification process, topological integrity and boundary preservation are maintained to ensure that the simplified mesh remains a true representation of the original structure. All reconstructed particle models are normalized to a uniform number of vertices and faces. This simplification is crucial because it provides a consistent basis for subsequent particle morphology evaluation and facilitates the import of the model into a discrete element model (DEM) for mechanical analysis and numerical simulation, such as... Figure 8 As shown.

[0103] Step (6): Statistical Analysis and Physical Model Output. For the statistical consistency assessment of particle size and shape, a distribution constraint and monitoring mechanism is built into the generation and reconstruction process: this includes the particle size distribution and shape indices (such as density) of the particle ensemble. elongation Flatness A reference distribution is established, and statistical measures such as KL divergence and JS divergence are used to quantitatively evaluate the statistical distribution of the reconstructed particle ensemble against the target distribution. This provides a basis for quality control and the effectiveness of the large particle model, and ensures the consistency and robustness of the reconstruction results in terms of ensemble statistical properties. Finally, the explicit geometric model and its structured statistical indices, after scale reduction and mesh simplification, are directly used as input parameters for the physical simulation model (DEM), including the geometric shape and distribution characteristics of the particles. The generated three-dimensional particle ensemble can cover the target particle size distribution (PSD), and based on this, statistically representative particle samples can be constructed, completing the automated process of "single image → three-dimensional reconstruction → statistical analysis → physical simulation".

[0104] The implementation of the various embodiments of the present invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of the present invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of the present invention provide a system for reconstructing three-dimensional digital particles based on a single grainy scene photograph. This system is used to execute a method for reconstructing three-dimensional digital particles based on a single grainy scene photograph as described in the above method embodiments.

[0105] See Figure 9 The system includes:

[0106] The particle segmentation module is used to input a 2D scene image containing multiple particles into a U-Net-assisted image segmentation framework, outputting a mask containing all particles in the image. Based on the mask, cropping and background replacement are performed to extract the normalized views of all individual particles in the image. The particle multi-view generation module is used to infer the normalized views of all individual particles one by one using a trained multi-view generation model to generate corresponding multi-view images. During the inference phase, the multi-view generation model uses the input normalized views of individual particles as conditions and employs the U-Net architecture to iteratively denoise in the latent space, gradually generating the latent representation of the multi-view image of individual particles from pure noise, and then reconstructing it into a multi-view image through a decoder. The particle 3D reconstruction module is used to input the multi-view images of all individual particles sequentially into a fine-tuned large reconstruction model to obtain the 3D mesh model of all individual particles. The reconstructed particle correction module is used to restore the physical scale of all the 3D mesh models of individual particles based on a scaling factor, and then perform mesh simplification to obtain the corresponding 3D digital particles of all individual particles.

[0107] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0108] The method in this embodiment of the invention is implemented using an electronic device; therefore, it is necessary to introduce the relevant electronic device. For this purpose, embodiments of the present invention provide an electronic device, such as... Figure 10 As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor invokes logical instructions stored in the at least one memory to execute all or part of the steps of the methods provided in the foregoing method embodiments.

[0109] Furthermore, when the logical instructions in at least one of the aforementioned memories are implemented as software functional units and sold or used as independent products, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, is embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various method embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks—various media for storing program code.

[0110] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0111] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0113] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0114] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0115] Based on the same technical concept as the foregoing embodiments, the present invention provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute a method for reconstructing three-dimensional digital particles based on a single grain scene photograph.

[0116] In summary, this invention discloses a method for 3D reconstruction of particle morphology based on a single photograph of a soil particle scene. This method addresses the problems of existing technologies, such as heavy reliance on multi-view acquisition, complex equipment, and insufficient generalization ability in new scenes, by proposing an end-to-end reconstruction process based on a combination of general and specialized visual models. The process includes: Step 1, using a U-Net-enhanced image segmentation framework to perform instance segmentation on a single 2D image of a particle set, obtaining the independent contour and mask of each particle; Step 2, for each particle, introducing a denoising diffusion probability model for multi-view synthesis, generating images from other perspectives based on a single view; Step 3, collecting digital particle samples, fine-tuning the general large-scale reconstruction model into a particle large-scale reconstruction model, using the multi-view images generated in Step 2 as input to the particle large-scale reconstruction model, and reconstructing 3D particle objects; Step 4, performing mesh sparsification and size correction on the generated 3D digital particles to obtain digital particles that can be directly used for discrete element method and other physical simulations. The results show that the proposed method can accurately predict particle size distribution and shape parameters, supports zero-shot cross-scene generalization, reduces dependence on hardware multi-view acquisition and specific training sets, and has advantages such as high automation, wide applicability, and low engineering deployment cost. It can be applied to fields such as water conservancy, civil engineering, mining, and bulk industrial process monitoring and analysis, and is used for three-dimensional reconstruction and morphology analysis of rock and soil mineral particles, crop particles, and other discrete granular materials. As an extension, this invention is also applicable to three-dimensional modeling, quality monitoring, and statistical distribution assessment of particulate targets in microscopic images, industrial inspection images, and remote sensing images.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph, characterized by, The method comprises the following steps: inputting a two-dimensional scene image containing multiple particles into a U-Net assisted image segmentation framework, outputting a mask containing all particles in the image, performing cropping and background replacement based on the mask, and extracting the standardized view of all single particles in the image; using a trained multi-view generation model to infer the standardized view of all single particles one by one to generate corresponding multi-view images; in the inference stage, the multi-view generation model uses the input single particle standardized view as a condition, adopts a U-Net architecture to iteratively denoise in the latent space, gradually generates a multi-view image latent representation of a single particle from pure noise, and reconstructs the multi-view image through a decoder; inputting all single particle multi-view images into a large reconstruction model after fine-tuning to obtain three-dimensional mesh models of all single particles; performing physical scale restoration on the three-dimensional mesh models of all single particles based on a scaling factor, and then performing mesh simplification to obtain three-dimensional digital particles corresponding to all single particles.

2. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 1, wherein, The U-Net assisted image segmentation framework processes the input image through the U-Net network to obtain a preliminary mask; the preliminary mask is converted into a point prompt, which is input into an image segmentation basic model together with the original input image; in the image segmentation basic model, an image embedding of the original input image is extracted through an image encoder; a prompt embedding of the point prompt is extracted through a prompt encoder; the image embedding and the prompt embedding are combined through a mask decoder, and a feature representation is updated using an attention mechanism, and then a multi-layer perceptron is used to map the output label to a feature vector, and then a spatial dot product operation is performed on the image embedding to generate a mask containing all particles in the image.

3. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 1, wherein, The training process of the multi-view generation model comprises: using a large-scale three-dimensional data set containing real multi-view images as a training data set; using a diffusion model framework to construct a multi-view image generation network; training the multi-view image generation network using the training data set, and the training process comprises: in the forward process, gradually adding Gaussian noise to the representation of the real single particle multi-view image in the latent space in the training data set at each time step to generate a series of intermediate noise latent representations until pure noise; in the backward inference process, using the input real single particle standardized view as a condition, using a U-Net architecture to iteratively denoise the pure noise at each time step to generate a set of representations of a single particle multi-view image in the latent space corresponding to a set of predefined camera poses, and reconstructing the single particle multi-view image through a decoder.

4. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 1, wherein, The large reconstruction model uses an encoder-decoder architecture to learn a three-dimensional representation of a single particle multi-view image: a visual encoder encodes the input image into a patch-level feature sequence, and a decoder combines camera parameters to generate an axis-aligned three-plane representation; for any three-dimensional sampling point, project it to the three planes for interpolation to obtain a feature vector, and input it into a shared multi-layer perceptron to parameterize a neural radiance field to represent the target object; a neural radiance field optimization pipeline with weight inheritance is used to integrate a differentiable isosurface extraction module to extract a triangular mesh surface from the parameterized neural radiance field to obtain a reconstructed three-dimensional model of a single particle.

5. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 4, wherein, The fine-tuning stage of the large reconstruction model freezes the encoding layer parameters, only updates the decoding layer and three-plane projection related parameters, uses the data in the fine-tuning training data set for multi-view supervision, and the fine-tuning training target is composed of pixel-level reconstruction loss, perception loss and mask consistency loss.

6. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 1, wherein, The scaling factor is selected according to the particle aspect ratio and the roundness threshold, and at least one of the following: 、 、 ; wherein, , and are scaling factors; is the projected area of the i-th single particle, is the maximum value of the projected areas of all single particles in the image; and are the maximum and minimum projected characteristic dimension of the i-th single particle, respectively, and are the maximum values of the corresponding maximum and minimum projected characteristic dimensions of all single particles in the image, respectively; n represents an empirical index value; is the circularity of the i-th single particle, is the maximum value of the circularities of all single particles in the image.

7. A method of reconstructing a three-dimensional digital particle based on a single particle scene photograph according to claim 1, wherein, The process of the mesh simplification is: based on the algorithm of the quadratic error metric, the three-dimensional mesh model of the single particle after the physical scale reduction is processed, and it is reduced to the target vertex number and the sheet number.

8. A system for reconstructing a three-dimensional digital particle based on a single particle scene photograph, characterized by, Comprise: The particle segmentation module is used for inputting a two-dimensional scene image containing multiple particles into a U-Net assisted image segmentation framework, outputting a mask containing all particles in the image, performing cropping and background replacement based on the mask, and extracting a standardized view of all single particles in the image; The particle multi-view generation module is used for using a trained multi-view generation model to infer all single particle standardized views one by one to generate corresponding multi-angle images; in the inference stage, the multi-view generation model uses the input single particle standardized view as a condition, uses the U-Net architecture to iteratively denoise in the latent space, gradually generates the multi-angle image latent representation of the single particle from pure noise, and is reconstructed into a multi-angle image by the decoder; The particle three-dimensional reconstruction module is used for inputting all single particle multi-angle images into the fine-tuned large reconstruction model in sequence to obtain three-dimensional mesh models of all single particles; The reconstructed particle correction module is used for performing physical scale reduction on the three-dimensional mesh models of all single particles based on the scaling factor, and then performing mesh simplification to obtain three-dimensional digital particles corresponding to all single particles.

9. An electronic device, comprising: The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method of any one of claims 1-7.

10. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method of any one of claims 1-7.

Images