A Method for RGB-D Nonmodal Completion and Particle Size Distribution Estimation of Densely Occluded Industrial Material Particles

By combining geometric and depth information from RGB-D images to separate adherent particles, identify occlusion states, and utilize a diffusion model to recover invisible areas and correct scale, the inaccuracy of particle measurement under dense occlusion conditions is solved, achieving more accurate particle size distribution estimation.

CN123118732APending Publication Date: 2026-11-17CENT SOUTH UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611379201.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-07
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies struggle to accurately separate adhering particles, identify occlusion states, restore invisible boundaries, acquire nonmodal monitoring data, and correct depth-related scales under conditions of dense occlusion, leading to inaccurate particle size distribution measurements.

Method used

By combining geometric and depth information from RGB-D images, adhering particles are separated and occlusion states are identified. The invisible areas of occluded particles are recovered using the structural priors of the diffusion model and RGB-D geometric constraints, and perspective scale correction and field-of-view edge compensation are performed.

Benefits of technology

It improves the reliability of particle separation and occlusion state identification under dense occlusion conditions, reduces the underestimation of particle size caused by occlusion, maintains the consistency of observed particle geometry, reduces reliance on manual annotation, reduces errors caused by stacking height and field of view edges, and improves the accuracy of particle size distribution estimation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118732A_ABST
    Figure CN123118732A_ABST
Patent Text Reader

Abstract

This invention discloses an RGB-D nonmodal completion and particle size distribution estimation method for densely occluded industrial material particles. It utilizes RGB-D information for particle instance separation and occlusion state identification, and automatically constructs occlusion training samples and nonmodal annotations from complete visible particles. A bi-branch stable diffusion nonmodal network is employed to fuse diffusion structure priors and RGB-D geometric constraints to recover the invisible regions of occluded particles. Unoccluded particles are then fused with the completed occluded particles, and perspective scale correction and field-of-view edge compensation are implemented using depth information to obtain the physically equivalent particle size and particle size distribution. This invention can reduce particle size measurement errors caused by particle adhesion, occlusion, and imaging scale changes under dense stacking conditions, improving the accuracy and reliability of particle size distribution estimation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of electronic information technology, specifically a method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles. Background Technology

[0002] Industrial particles are widely present in industrial processes such as mining, metallurgy, chemical industry, building materials, and material processing. Particle size distribution is an important indicator reflecting the state of materials, process stability, and product quality. With the development of machine vision, depth sensing, and deep learning technologies, using visual information for non-contact particle size measurement of industrial particles has become an important technical approach in addition to traditional mechanical screening and sedimentation analysis.

[0003] Based on the current research status in fields such as industrial particle visual measurement, RGB-D perception, occluded target completion, and visual particle size calculation, existing technologies that are similar to this invention mainly include the following categories: 1. Particle segmentation and particle size measurement schemes based on traditional image processing: Early visual particle size measurement methods mainly utilize the grayscale, edges, and geometric shapes of particles in two-dimensional images for processing. Typically, threshold segmentation, edge detection, distance transformation, morphological processing, or watershed algorithms are first used to obtain particle regions. Then, geometric parameters such as projected area, equivalent diameter, maximum inscribed circle, and Feret diameter are used to estimate particle size, and particle size distribution is statistically analyzed based on the number of particles or size ranges. For particle adhesion problems, existing methods typically use distance transformation, seed point extraction, watershed segmentation, convex hull or convex hull analysis, etc., for particle size analysis of coal particles, using a gradient-based watershed algorithm for initial segmentation, and combining small region merging and convex hull methods to handle adhered particles. Other studies have extracted dimensional features such as particle length, width, and maximum inscribed circle using machine vision and established a correspondence with mechanical sieving results for particle size estimation under conveyor belt or experimental conditions. This type of method is simple to implement, has relatively low computational cost, and is applicable when particles are dispersed, have clear outlines, and relatively regular shapes. 2. Deep Learning-Based Industrial Particle Visual Measurement Scheme: With the development of deep learning, U-Net, Mask R-CNN, and their improved networks have been increasingly applied to semantic segmentation or instance segmentation of ores, coal, rocks, and other industrial particles. Semantic segmentation methods typically utilize convolutional neural networks to directly predict particle regions and background regions. For example, existing research has applied U-Net to mineral particle size analysis in conveyor belts and improved the segmentation effect of dense particles by explicitly adding particle boundary information. Instance segmentation methods output independent masks for each particle while detecting its position, reducing confusion between adjacent particles using models such as Mask R-CNN. The obtained particle masks can be further used to calculate equivalent diameter, projected area, or other geometric parameters to obtain the visual particle size distribution. Compared to traditional image processing methods, deep learning methods can learn complex texture and shape features and are more adaptable to changes in lighting, background interference, and differences in particle appearance. 3. Particle perception scheme based on RGB-D multimodal information: When using only RGB images, the boundaries between adjacent particles are easily confused when industrial particles have similar colors, textures, or exhibit shadows, reflections, and surface texture variations. To improve spatial perception capabilities in complex scenes, existing visual perception methods are gradually incorporating depth information, fusing RGB appearance features with depth geometric information.

[0004] RGB-D methods typically utilize RGB images to represent color, texture, and local boundaries, while using depth maps to provide surface height, spatial distance, and spatial relationships. Existing RGB-D semantic segmentation and scene awareness methods employ convolutional feature fusion, attention mechanisms, or Transformer structures to jointly model the two modalities, thereby improving the ability to distinguish between similar-looking targets.

[0005] In industrial particle scenarios, depth information can also be used to identify height differences on particle surfaces, assist in locating the center of adhered particles, and determine the front-to-back relationship between adjacent particles. Therefore, compared to using RGB information alone, RGB-D perception has a stronger spatial geometric representation capability. 4. Complete Shape Recovery Scheme Based on Non-modal Completion of Occluded Targets: Existing two-dimensional semantic segmentation or instance segmentation methods typically only describe the currently observable visible area of ​​an object. When an object is occluded by other objects, its output mask does not include the occluded part. To address this problem, the field of computer vision has proposed non-modal perception and non-modal completion techniques to infer the complete shape of the target from the partially visible area. Early methods mainly utilized contour continuity, convexity assumptions, parametric geometric models, or predefined shape priors to recover invisible areas. Subsequently, learning methods based on convolutional neural networks, encoder-decoders, generative adversarial networks, and shape codebooks have been used to predict the complete object mask from the visible mask or local image. For example, some non-modal methods recover hidden areas by learning the correspondence between complete objects and occluded objects; other methods provide a complete shape reference for predicting invisible areas by constructing a discrete shape codebook. These techniques provide a foundation for recovering the complete geometry of an object from partial visual observations. 5. Occlusion target completion scheme based on diffusion model priors:

[0006] In recent years, diffusion models have demonstrated strong data distribution modeling capabilities in image generation and visual representation learning. Their denoising networks, during pre-training, can form multi-layered latent representations encompassing the overall shape organization, contour continuity, and spatial relationships of objects, thus gradually being used for non-modal target recovery. Existing research directly utilizes the representation capabilities of pre-trained StableDiffusion models for non-modal segmentation, while other methods leverage diffusion models to extract shape priors and predict the complete shape of occluded regions. For example, AISDiff extracts visual features from conditional diffusion models for non-modal instance segmentation through a diffusion shape prior estimation module; other studies employ diffusion inpainting models to generate the complete appearance of targets from partially visible images. These methods demonstrate that the latent representations in pre-trained diffusion models can provide richer overall structural information for hidden region inference than local convolutional features. 6. Granularity Calculation Scheme Based on Visual Geometric Parameters: After obtaining the granular segmentation mask, existing visual granularity measurements typically calculate granular size using parameters such as pixel area, equivalent diameter, maximum inscribed circle, length, or width, and then convert the pixel size into actual physical size based on camera calibration results. Among them, the Heywood equivalent diameter can be calculated based on the projected area of ​​the particle, yielding an equivalent circle diameter with the same area, suitable for uniform size characterization of irregular particles. Some visual particle size measurement methods also obtain quantitative or qualitative particle size distributions by statistically analyzing particle number, particle size range, or performing mass-equivalent weighting based on particle size cube. Under conditions of fixed shooting distance and relatively flat material layer, a uniform pixel-physical size calibration coefficient can generally be used to complete the size conversion.

[0007] While the aforementioned methods can achieve industrial particle identification, segmentation, occlusion recovery, and size calculation to a certain extent, they still have the following shortcomings in measuring the particle size distribution of densely stacked industrial materials: 1. Traditional image processing methods are insufficiently adaptable to densely adhered and irregular particles: Particle separation methods based on distance transformation, morphology, watersheds, and geometric fitting typically rely on relatively clear two-dimensional boundaries or relatively regular geometric shapes. When particles have irregular shapes, surface pores, grooves, or strong adhesion, the extreme positions of the distance field are prone to deviating from the true particle center. Simultaneously, industrial particle surfaces often exhibit rough textures, shadows, and local brightness variations, making incorrect segmentation easily generated by relying solely on two-dimensional images. Directly using depth maps for watershed processing is also susceptible to oversegmentation due to particle surface undulations and depth sensor noise.

[0008] Therefore, neither single 2D geometric information nor single depth information can reliably separate densely clustered particle instances. 2. Existing semantic segmentation and instance segmentation methods can only obtain the visible region of particles: Deep learning segmentation methods such as U-Net and Mask R-CNN can improve the ability to extract particle regions, but their training targets are usually the visible regions in the current image. For instances located in the lower layer and covered by other particles, the network can only output the contours that are still exposed in the field of view, and cannot directly recover the completely invisible boundaries. Therefore, the obtained particle mask is not equivalent to the real and complete geometric structure of the particles. In particle size measurement, the particle size is usually calculated directly based on the mask area or boundary, so the lack of hidden contours can easily lead to an underestimation of particle size; if adjacent particles fail to de-adhere correctly, multiple particles may be merged for calculation, further affecting the statistical results of particle size distribution. 3. RGB-D information alone cannot obtain completely hidden regions: RGB-D data can use depth differences to assist in particle separation and occlusion status judgment, but depth sensors, like RGB cameras, can only acquire the surface that is actually visible from the current viewpoint. For regions completely covered by upper-layer particles, their true depth and boundaries cannot be directly observed. Therefore, even if the RGB-D method can determine that a particle is occluded, it cannot directly recover its complete geometry solely based on sensor measurements. This means that RGB-D perception can provide "currently observed geometric constraints," but still requires additional structural priors to infer completely invisible areas. 4. Existing non-modal completion methods are insufficient for matching the geometric features of industrial particles: Most existing non-modal completion methods are designed for people, vehicles, animals, and everyday objects in natural images. These targets usually have relatively clear semantic categories and relatively stable overall shapes. Industrial material particles, on the other hand, have weak semantic information, highly irregular shapes, obvious non-convexity, and large individual differences, making them difficult to accurately describe using fixed shape templates or finite discrete shape codebooks. Therefore, directly using existing non-modal completion methods for natural images can easily lead to problems such as over-regularization of hidden contours, discontinuous connections between local contours and visible boundaries, and completion results that cannot reflect the true morphological changes of particles. 5. Directly using diffusion models to generate results is difficult to meet quantitative measurement requirements: Pre-trained diffusion models can provide richer visual and structural priors, but their main goal is usually to obtain visually reasonable image generation or restoration results, rather than strictly maintaining the actual geometric dimensions of the input target. If a complete particle is directly generated using a diffusion model, it requires multiple reverse denoising iterations, increasing computational overhead. Furthermore, the generated result may alter the observed particle boundaries or introduce hidden shapes inconsistent with the actual depth. For particle size estimation, the particle boundaries directly determine the projected area and equivalent diameter; therefore, generating results that only pursue visual plausibility are unlikely to meet the geometric consistency requirements of subsequent quantitative size measurements.Furthermore, existing diffusion nonmodal methods are mainly designed for natural image targets and have not established a joint mechanism between diffusion structure information and actual RGB-D geometric constraints for industrial particles. 6. Lack of reliable nonmodal supervision data for real occluded areas: The complete boundaries of occluded particles cannot be directly observed in real RGB-D images. Therefore, when constructing nonmodal training data for industrial particles, manual annotators need to subjectively infer invisible areas. This annotation method is not only labor-intensive, but different annotators may also have different judgments on the hidden particle contours, making it difficult to form stable and consistent large-scale supervision data. Therefore, how to automatically construct occluded samples that conform to RGB-D spatial relationships using actually visible complete particles, and simultaneously obtain a definite complete mask as supervision, is also a problem that existing industrial particle nonmodal completion methods need to solve. 7. Fixed-scale calibration is difficult to adapt to the height variation of densely stacked particles: Existing visual particle size measurement usually uses a uniform pixel-to-physical size conversion coefficient when the camera position is fixed. However, densely stacked particles have different surface heights, and the actual distance from different particles to the camera is not the same. According to perspective imaging relationships, even if two particles have the same actual physical size, their pixel size in the image will be different as long as their imaging distance is different. Therefore, a uniform scaling factor can easily introduce systematic size biases related to grain depth. 8. Truncation of grains at the field of view edge can easily lead to grain size underestimation: For grains near the edge of the camera's field of view, part of their area may be outside the image range, resulting in the currently obtained grain mask area being smaller than its full projected area. If the area-equivalent diameter calculation method for full grains is still used, the size of such grains will be underestimated, further altering the number or mass ratio of grains in the corresponding grain size range. Existing visual grain size measurement methods typically discard such grains directly or still calculate based on incomplete area, making it difficult to fully utilize the information from grains at the field of view edge.

[0009] In summary, existing industrial particle visual particle size measurement technologies are still insufficient to simultaneously address issues such as the separation of densely adhered particles, identification of occlusion states, restoration of invisible boundaries, automatic generation of non-modal supervisory data, and depth-related physical scale correction within the same technical process. Therefore, this invention aims to provide an RGB-D non-modal completion and particle size distribution estimation method for densely occluded industrial material particles. This method can utilize RGB-D information to identify the actual geometric relationships of particles, further introduce structural priors applicable to irregular industrial particles to restore invisible regions, while maintaining the observed geometric structure unchanged, and perform perspective scale correction and field-of-view edge compensation on the reconstructed complete particles. Summary of the Invention

[0010] The purpose of this invention is to provide an RGB-D nonmodal completion and particle size distribution estimation method for densely occluded industrial material particles, in order to solve problems such as the difficulty in accurately separating adhered particles under dense stacking conditions, the difficulty in reliably identifying the occlusion state, the inability to directly observe the hidden boundaries of occluded particles, the lack of effective structural constraints in the hidden regions of industrial material particles, the difficulty in obtaining nonmodal supervision data, the inaccuracy of uniform scale conversion caused by differences in particle stacking height, and the underestimation of particle size due to particle truncation at the edge of the field of view.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] This invention provides a method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles, comprising the following steps:

[0013] Step 1: RGB-D Particle Data Acquisition and Visible Region Extraction: Acquire RGB and depth images that are synchronously collected and spatially registered in the industrial material particle scene, so that the pixel positions in the RGB image correspond to the depth measurement positions in the depth image; and obtain the visible region mask of the industrial material particles based on the RGB image and / or depth image.

[0014] Step 2: Separation of Adhesive Particle Instances Based on Geometric and Depth Information: For an adhesive connected region formed by multiple particles in the image space, a geometric potential field is constructed based on the distance from the pixels inside the connected region to the boundary, and a depth potential field is constructed based on the depth data corresponding to the connected region; the geometric potential field and the depth potential field are fused to construct a fused topographic map, the particle instance marker positions are determined based on the fused topographic map, and the adhesive connected region is segmented using the particle instance marker positions to obtain mutually independent particle instance masks;

[0015] Step 3: Occlusion state recognition based on particle boundary depth relationship: For each particle instance that has been separated, local depth information is obtained on the inside and outside of the particle boundary respectively. Based on the depth difference inside and outside the particle boundary, it is determined whether the particle has an occlusion boundary covered by other particles.

[0016] When there are other particle surfaces on the outside of the particle boundary that are closer to the RGB-D vision acquisition device than the inside of the particle, and the depth difference exceeds the preset threshold used to suppress the influence of depth measurement noise, the corresponding boundary is determined to be an occlusion boundary.

[0017] Based on whether there is an effective occlusion boundary, particle instances are divided into unoccluded particles and occluded particles;

[0018] Step 4: Automatic generation of multimodal occlusion training samples and nonmodal supervised annotations: Select at least two complete visible particles from the set of unoccluded particles, one as the base particle and the other as the occluded particle; perform at least one spatial transformation on the occluded particle, including rotation, scaling and translation, so that the transformed occluded particle and the base particle have effective spatial overlap.

[0019] Based on the depth information corresponding to the base particles and the occluded particles, the transformed occluded particles are depth aligned so that the occluded particles have a depth relationship in front of the base particles in the overlapping area; then the RGB information, depth information and particle mask corresponding to the two particles are combined synchronously to form RGB-D occlusion training samples with consistent front and back occlusion relationships.

[0020] The complete mask before the basic particle is occluded is used as the non-modal supervision annotation of the training sample, and the remaining visible area after the basic particle is occluded is used as the incomplete visible mask.

[0021] Step 5, Nonmodal Completion Network Training: Train a bi-branch nonmodal completion network using the RGB-D occlusion training samples obtained in Step 4. The bi-branch nonmodal completion network includes a diffusion structure prior extraction branch and an RGB-D geometry guidance branch.

[0022] Step 6, Non-modal Completion of Occluded Particles: The RGB local image corresponding to the occluded particle to be completed is input into the diffusion structure prior extraction branch. Intermediate latent features at different levels are extracted as diffusion structure priors through a pre-trained diffusion model. The incomplete visible mask of the particle to be completed and its corresponding local depth map are input into the RGB-D geometry guidance branch. RGB-D geometric features are obtained through convolutional encoding. The diffusion structure prior and RGB-D geometric features are fused at corresponding feature scales, and the initial non-modal prediction result of the occluded particle is obtained through decoding.

[0023] Step 7, Visible Area Preservation Constraint: The initial non-modal prediction result is constrained by the incomplete visible mask of the particle to be completed, so that the particle area already observed by RGB-D visual data remains unchanged, and a complete non-modal particle mask containing the visible area of ​​the particle and the recovered invisible area is obtained.

[0024] Step 8: Complete Particle Instance Fusion: For unoccluded particles obtained through occlusion state recognition, retain their original complete visible masks; for occluded particles, use the complete non-modal mask obtained by the non-modal completion network; reorganize the unoccluded particles and the occluded particles that have completed non-modal completion according to their spatial positions in the original image to obtain a complete set of particle instances.

[0025] Step 9, perspective scale correction: For each particle in the complete set of particle instances, extract effective depth data from its corresponding depth region to determine the representative imaging depth, and combine the representative imaging depth and camera calibration parameters to determine the scale transformation relationship of the corresponding particle from pixel size to physical size.

[0026] Step 10, Calculation of Particle Physical Size and Compensation for Field of View Edge: Determine whether the particles intersect with the image field of view boundary. For particles that do not intersect with the field of view boundary, calculate the physical equivalent particle size based on the complete particle mask and the corresponding scale transformation relationship. For truncated particles that intersect with the field of view boundary, estimate the equivalent size based on the boundary within the effective field of view, and obtain the compensated physical equivalent particle size by combining the corresponding scale transformation relationship.

[0027] Step 11, Particle size distribution calculation: Based on the physical equivalent particle size of each effective particle, the quantity is statistically analyzed according to the preset particle size range. Under the condition that the differences in particle material density and shape factor of similar industrial materials can be approximately ignored, the mass equivalent weighting is performed based on the cube of the physical equivalent particle size to obtain the mass equivalent particle size distribution of the industrial material particles.

[0028] As a further improvement, the geometric potential field in step two is established by performing an Euclidean distance transformation on the binary mask and establishing a normalized geometric potential field based on the distance from each pixel within the connected region to the background boundary; the depth potential field is obtained by extracting the depth region corresponding to the connected region from the depth map and performing smoothing and normalization processing; the fused terrain map is constructed as follows:

[0029] ;

[0030] in, It is a geometric potential field; For deep potential field, and These are the weights corresponding to the two types of potential fields. Represents element-wise product. It is a matrix of all ones with the same size as the input mask.

[0031] As a further improvement, in step three, the pixel positions on the particle boundary are selected along the boundary normal direction, with the inner and outer neighborhoods of the particle being selected respectively, and the local depth gradient at the boundary position is defined as:

[0032] ;

[0033] in, The average depth of the neighborhood inside the boundary. The average depth of the neighborhood outside the boundary; when the region of a continuous boundary satisfies: At that time, among them, The boundary is determined as an occlusion boundary by a threshold used to suppress the influence of noise in depth sensor measurements.

[0034] As a further improvement, the depth alignment in step four includes: calculating the depth offset based on the depth statistics of the base particle and the occluded particle in the corresponding area, and introducing a positive depth offset to keep the occluded particle in front of the base particle in the overlapping area.

[0035] As a further improvement, the diffusion structure prior extraction branch in step five adopts a pre-trained StableDiffusion model and extracts its U-Net multi-scale intermediate latent features at different encoding stages as structure priors; the U-Net parameters of the pre-trained StableDiffusion model are kept frozen during the training of the non-modal completion network.

[0036] As a further improvement, in step seven, the visibility mask preservation constraint is applied using the initial non-modal prediction results obtained in step six in the following manner:

[0037] ;

[0038] in, This is the initial predicted probability map. This is an incomplete visibility mask for the particles. The output is the result after maintaining constraints through the visible region. This represents element-wise product.

[0039] As a further improvement, in step five, the training loss of the bi-branch nonmodal completion network consists of binary cross-entropy loss and Dice loss:

[0040] ;

[0041] in, For binary cross-entropy loss, For Dice's loss, and These are the weights corresponding to the two loss terms.

[0042] As a further improvement, step nine... The physical scale conversion factor of each particle is determined as follows: ;

[0043] in, For the first The scale transformation coefficient of each particle. This represents the representative imaging depth of the particle. The focal length parameters obtained from camera calibration These are calibration parameters related to the sensor pixel scale.

[0044] As a further improvement, in step ten, for truncated particles intersecting the image field of view boundary, the effective particle boundary points that are still inside the image are extracted, and the equivalent radius corresponding to the particle is estimated by least-squares circle fitting. The physical equivalent diameter after truncation compensation is calculated as follows: ;

[0045] in, The pixel equivalent radius obtained by fitting edge particles. This is the scale transformation factor for the particle. The equivalent diameter of the particle after field-of-view truncation compensation.

[0046] As a further improvement, the particle size distribution in step eleven includes quantitative particle size distribution and mass-equivalent particle size distribution; for mass-equivalent particle size distribution, the first step is calculated as follows: Equivalent mass percentage of each particle size range:

[0047] ;

[0048] in, For the first The number of particles in each particle size range This refers to the total number of particles involved in particle size statistics. The first [value] obtained after depth scale correction and necessary field-of-view edge compensation Physical equivalent diameter of each particle For the first to participate in granularity statistics The physical equivalent diameter of each particle is obtained after perspective scale correction and necessary field-of-view edge compensation.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] 1. Improve the reliability of instance separation and occlusion state identification of adherent particles under densely stacked conditions. This invention combines the geometric potential field formed by the two-dimensional contour of the particles with the depth potential field formed by the depth surface information for the separation of adherent particles. This allows the two-dimensional geometric information and three-dimensional depth information to complement each other, reducing the instability of irregular particle center positioning when relying solely on distance transformation, and mitigating missegmentation caused by particle surface undulations and depth noise when directly relying on the depth surface. Based on instance separation, the occlusion state is further determined according to the depth relationship inside and outside the particle boundary, enabling occluded particles that require hidden region recovery to be distinguished from complete visible particles, thereby providing more reliable target instance and occlusion state information for subsequent non-modal completion.

[0051] 2. This invention can recover the hidden boundaries of occluded particles that traditional visible region segmentation methods cannot directly obtain, thereby reducing the underestimation of particle size caused by occlusion. Existing semantic segmentation or instance segmentation methods can usually only obtain the visible region of particles from the current viewpoint, and cannot directly recover the parts covered by other particles. This invention performs non-modal completion for occluded particles, combining the overall structural information provided by the diffusion model with the RGB-D local geometric constraints of the current particle, so that the recovery of the hidden region depends not only on the local visible contour, but also on the actual depth relationship of the particle and the visible boundary.

[0052] 3. Maintaining consistency of observed particle geometry while utilizing diffusion model structural information. This invention does not directly generate complete particle images using the diffusion model. Instead, it extracts multi-scale latent structural features from a frozen pre-trained diffusion network to provide information on the overall shape organization and contour continuity of the particles. Simultaneously, it encodes the visibility mask and depth information of the current particle through independent RGB-D geometric branches and performs cross-modal fusion during the decoding stage. Furthermore, a visibility mask preservation constraint is set to directly retain the particle regions already observed by the sensor, while the network primarily predicts invisible regions.

[0053] 4. This invention enables the automatic acquisition of non-modal training supervision using real, complete particles, reducing the reliance on manual annotation of hidden regions. For real occluded particles, their invisible boundaries cannot be directly obtained by RGB-D sensors, requiring manual annotation to subjectively infer the hidden contours. This invention utilizes identified unoccluded complete particles to construct RGB-D training samples with before-and-after occlusion relationships through spatial transformation, particle overlap, and local depth alignment, directly retaining the mask of the complete particle before occlusion as non-modal supervision annotation. Therefore, the synthesized incomplete visible observations and complete particle annotations can be automatically generated from the same real particle, thereby reducing the reliance on manually drawing invisible boundaries and providing consistent training supervision for the non-modal completion model.

[0054] 5. Reduce errors in actual particle size calculation caused by variations in particle stacking height and field-of-view truncation. After obtaining the complete particle mask, this invention does not use a uniform pixel-to-physical size conversion coefficient across the entire field of view. Instead, it utilizes the depth information corresponding to each particle to determine the scale conversion relationship, thereby compensating for perspective scale variations caused by different stacking heights. For particles intersecting with image boundaries, the equivalent size is estimated using boundary information still valid within the field of view, reducing particle size underestimation caused by incomplete projected area. This processing allows the complete image spatial geometry obtained through non-modal completion to be further converted into geometrically calibrated physical particle sizes, providing a more reliable dimensional basis for particle size distribution calculation.

[0055] 6. Improve the accuracy of particle size distribution estimation for densely obscured industrial materials. In the complete measurement process, particle instance separation, non-modal completion, perspective correction, and field-of-view edge compensation are applied sequentially to different error sources, thereby reducing the cumulative bias that occurs when particle size is calculated directly based solely on the visible particle area. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the structure of the industrial material particle RGB-D data acquisition system of the present invention;

[0057] Figure 2 This is a schematic diagram illustrating the problem of particle occlusion and boundary loss in densely stacked industrial materials addressed by this invention, wherein: Figure 2 (a) in the image is the original RGB image of the industrial material particles; Figure 2 (b) in the image is the depth image after registration with the RGB image space; Figure 2 (c) in the image is the particle mask result obtained by the traditional visible region segmentation method; Figure 2 (d) in the diagram is a schematic diagram of the non-modal true boundary of the complete particle structure;

[0058] Figure 3 The flowchart of the overall method for RGB-D nonmodal completion and granular distribution estimation based on diffusion structure prior guidance proposed in this invention shows the complete processing flow from RGB-D data input, particle perception, occlusion state recognition, nonmodal completion, instance fusion to physical granularity calculation.

[0059] Figure 4 This is a schematic diagram of the separation process of adhering particles based on depth information in this invention, wherein: Figure 4 (a) in the image represents the original RGB image and the extreme points of the particle region; Figure 4 (b) in the image is the corresponding 3D depth map; Figure 4 (c) in the image represents the initial mask of the adhered particles; Figure 4 (d) in the figure represents the topological response map and instance seed point map formed by combining geometric and depth information; Figure 4 (e) in the image is the mask image of the separated individual particle instances;

[0060] Figure 5 This is a schematic diagram of the occlusion state recognition process based on particle boundary depth changes in this invention, wherein: Figure 5 (a) in the image is the result of overlaying the particle mask with the RGB image; Figure 5 (b) in the image is the corresponding depth pseudo-color image; Figure 5 (c) in the diagram is a schematic diagram of the depth gradient direction at the particle boundary; Figure 5 (d) in the figure represents the particle occlusion classification result obtained based on the boundary depth relationship;

[0061] Figure 6 This is a schematic diagram illustrating the automatic generation process of occlusion training samples and non-modal supervised annotations based on multimodal image transformation in this invention. Figure 6 (a) is a group of particles sampled from intact visible particles; Figure 6 (b) is the result of spatial transformation of the sampled particles to form a simulated visual occlusion; Figure 6 (c) is the result of depth alignment based on the depth relationship between particles and depth maps after fusion. Figure 6 (d) is the process of extracting target particle tags from synthetic occlusion samples; Figure 6 (e) is the modal visibility mask after the target particle is occluded; Figure 6 (f) is the complete non-modal supervision annotation corresponding to the target particle before occlusion;

[0062] Figure 7 This is a schematic diagram of the dual-branch stable diffusion nonmodal network (DSDA-Net) structure based on diffusion structure prior and RGB-D geometric constraints in this invention;

[0063] Figure 8 This is a schematic diagram illustrating the process of calculating the particle size of industrial materials based on depth scale correction and field-of-view edge compensation in this invention. Figure 8 (a) shows the global particle segmentation result and the field-of-view edge particle recognition result. Figure 8 (b) shows the calculation process for the Heywood equivalent diameter of the internally intact particles. Figure 8 (c) is the process of using least squares circle fitting to compensate for the size of truncated particles at the edge of the field of view and to calculate the equivalent diameter. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0065] This invention provides an RGB-D nonmodal completion and particle size distribution estimation method for densely occluded industrial material particles. Addressing the size measurement errors caused by particle adhesion, mutual occlusion, missing visible boundaries, and truncation at different stacking heights and field-of-view edges in densely stacked industrial material particles, this invention utilizes complementary information between RGB and depth images to establish a complete technical flowchart consisting of particle perception, occlusion reconstruction, instance fusion, and geometric calibration particle size estimation.

[0066] like Figure 2As shown, in densely stacked industrial material particles, RGB images can reflect the surface texture and appearance information of the particles, and registration depth maps can reflect the spatial height and hierarchical relationship of the particle surface. However, conventional visible area segmentation results can only obtain the unoccluded particle area. When the lower layer particles are covered by the upper layer particles, some of their true boundaries cannot be directly observed, resulting in the segmented particle area being smaller than its actual complete projected area. Therefore, this invention does not directly use visible particle masks for particle size measurement, but first determines the independent particle instances and their occlusion states, restores the complete geometry of the occluded particles, and then calculates the physical particle size.

[0067] like Figure 3 As shown, the overall technical framework of this invention consists of a perception stage, a reconstruction stage, and a fusion stage, which further perform geometric calibration and particle size distribution estimation based on the fused complete particle instances. The perception stage is responsible for separating mutually adhered particle instances from RGB-D observations and identifying the occlusion state of each particle; the reconstruction stage only targets occluded particles, using structural priors extracted from the diffusion model and geometric constraints provided by RGB-D observations to recover their invisible regions; the fusion stage recombines unoccluded particles and reconstructed occluded particles to obtain a complete set of particle instances; finally, perspective scale correction is performed according to the corresponding depth of each particle, and size compensation is performed on truncated particles at the edge of the field of view to obtain the particle size and particle size distribution at the actual physical scale.

[0068] The specific technical solution is as follows.

[0069] I. RGB-D particle sensing and visible area acquisition:

[0070] An RGB-D vision acquisition device is installed above the area where industrial material particles are conveyed or piled up to simultaneously acquire RGB and depth images of the particle scene. The RGB and depth images are then spatially registered to ensure that the color and depth information at the same image location have a corresponding relationship.

[0071] Let the acquired RGB image be The corresponding depth map is First, a particle segmentation method is used to obtain the visible region mask of industrial material particles in the current field of view. This preliminary particle segmentation method is not limited to a specific network structure, as long as it can obtain the visible region segmentation result between the industrial material particles and the background. Therefore, the subsequent occlusion perception and non-modal completion method of this invention can be combined with different preliminary particle segmentation methods. For particle regions that are separated from each other, independent particle instances can be directly established; for connected regions formed by multiple particles in a two-dimensional image, depth information and two-dimensional geometric information are further used for instance separation.

[0072] II. Separation of Adhesive Particles and Identification of Occlusion Status Based on Depth Prior:

[0073] The purpose of the perception phase is not to recover the invisible particle structure, but to answer two questions: first, how many independent particles are contained in the current adhesion region; and second, whether each separated particle is obscured by other particles.

[0074] This provides a basis for subsequently determining which particles need to undergo nonmodal reconstruction.

[0075] 1. Separation of adhering particle instances by combining geometric and depth information:

[0076] For any binary connected region of adherent particles First, a geometric potential field is constructed based on the two-dimensional contour of the region. Then, a binary mask is used. Perform a Euclidean distance transformation to establish a normalized geometric potential field based on the distances from each pixel within the connected region to the background boundary. The interior of the particle exhibits a higher geometric response at locations farther from the boundary, thus this potential field can characterize the potential particle center location. Meanwhile, from the depth map... Extracting connected components The corresponding depth region is then smoothed and normalized to construct the depth potential field. .

[0077] Since the RGB-D sensor acquires industrial material particles from top to bottom, under these imaging conditions, areas with smaller depth values ​​correspond to particle surfaces closer to the sensor. Therefore, the depth potential field can provide particle surface height information that cannot be directly obtained from a two-dimensional mask. By combining the geometric potential field and the depth potential field, a fused topographic map for instance separation is constructed. :

[0078] ;

[0079] in, It is a geometric potential field; For deep potential field, and These are the weights corresponding to the two types of potential fields. Represents element-wise product. It is a matrix of all ones with the same size as the input mask.

[0080] The marker locations required for watershed segmentation are determined based on the local structure in the fused topographic map. Then, the marker-based watershed method is used to divide the cohesive connected regions, thereby obtaining mutually independent particle instances.

[0081] This technology does not rely solely on two-dimensional distance information or solely on depth information. Instead, it uses two-dimensional contours to constrain the spatial range of instances and utilizes the difference in elevation between depth and surface to correct the location of instance centers under irregular particle conditions, thereby improving the stability of particle instance separation under dense adhesion conditions.

[0082] 2. Occlusion state recognition based on depth relationship inside and outside the boundary:

[0083] After obtaining independent particle instances, depth variation analysis is performed on the boundary region of each particle.

[0084] For each pixel location on the particle boundary, both the inner and outer neighborhoods of the particle are selected along the boundary normal direction, and the gradient is calculated. The local depth gradient at the boundary location is defined as:

[0085] ;

[0086] in, The average depth of the neighborhood inside the boundary. The average depth of the neighborhood outside the boundary.

[0087] In top-down imaging, if the current particle is a fully visible particle in the upper layer, its outer boundary usually corresponds to the background or the surface of other particles below it. Therefore, within the allowable measurement noise range, the following generally applies: If a continuous boundary region satisfies: ,in, If the threshold used to suppress the influence of noise in depth sensor measurements is set, it indicates that there are other particles outside the boundary that are closer to the sensor than the current particle, thus classifying the boundary as an occlusion boundary formed by other particles covering it.

[0088] Based on whether particles have effective occlusion boundaries, independent particles are classified into unoccluded particles (UP) and occluded particles (OP). For UP, the currently visible region can be used to represent its complete particle region, eliminating the need for hidden region prediction and allowing direct entry into subsequent instance fusion. OP, due to the existence of invisible regions, requires input to a non-modal completion module to recover its complete particle structure. This classification result is also used to build subsequent training data, specifically by automatically constructing occluded training samples using UP as reference particles with complete realistic shapes.

[0089] III. Generation of multimodal occlusion samples and nonmodal supervision based on complete visible particles:

[0090] In real-world industrial scenarios, occluded particles can only be observed at their exposed portions, making it impossible to directly obtain the true mask for invisible areas from sensors. Therefore, this invention avoids manually drawing hidden contours and instead utilizes the complete visible particles (UP) identified during the perception phase to establish non-modal supervised samples.

[0091] Select two complete particles from the UP set, one as the base particle (Base UP) and the other as the occluding particle (Top UP). Apply constrained spatial transformations to Top UP, including rotation, scaling, and translation, to move Top UP to the vicinity of Base UP and achieve effective spatial overlap between the two.

[0092] Let the basic particle mask be: The upper-layer particle mask after spatial transformation is: Then the overlapping region of the two particles is:

[0093] ;

[0094] When satisfied At this point, it is assumed that an effective occlusion relationship exists between the two particles. Since directly superimposing the RGB and depth data of two real particles may result in an unreasonable depth order in the overlapping area, local depth alignment of the Top UP is also required.

[0095] The depth offset is calculated based on the depth statistics of the Base Up and Top Up within the corresponding regions, and a small positive depth offset is introduced. This ensures that the Top Up remains in front of the Base Up within the overlapping area, thus creating a reasonable foreground-background depth relationship.

[0096] After spatial transformation and depth alignment, the RGB images, depth maps, and masks corresponding to Top UP and Base UP are simultaneously combined to form simulated occlusion samples with consistent RGB-D spatial relationships. The complete mask of Base UP before occlusion is used as the non-modal supervision annotation for the corresponding sample. The Base Up region, which remains visible even after being obscured by Top Up, serves as an incomplete visibility mask.

[0097] ;

[0098] Therefore, for each synthesized sample, a synthesized RGB image, a synthesized depth map, and an incomplete visible mask can be obtained simultaneously. And its corresponding complete nonmodal mask .

[0099] Therefore, the nonmodal supervised training data required for DSDA-Net can be generated without manually speculating the boundaries of hidden particles.

[0100] IV. Particle nonmodal completion based on diffusion structure priors and RGB-D geometric constraints:

[0101] For the occluded particles (OPs) identified during the perception phase, this invention employs a dual-branch stable diffusion nonmodal network (DSDA-Net) to recover their invisible regions. For example... Figure 7 As shown, DSDA-Net includes parallel diffusion structure prior branches, RGB-D geometry-guided branches, and multi-scale post-fusion decoding modules and visible mask physical consistency constraint modules located after the two branches. This invention does not use Stable Diffusion as a complete image generator, but instead utilizes the multi-scale latent features within its pre-trained U-Net as prior information for hidden particle structure recovery. The network employs a dual-branch encoding and multi-scale post-fusion structure. The diffusion structure prior branch takes the RGB local image corresponding to the target particle as input and extracts multi-scale structural features from the pre-trained diffusion model. The RGB-D geometry-guided branch takes the incomplete visible mask of the target particle and its corresponding depth map as input and extracts local geometric features related to the actual observation of the current particle. The two types of features are fused according to their corresponding scales during the decoding stage, and the initial non-modal prediction result is obtained through step-by-step decoding. Subsequently, the input visible mask is used to maintain constraints on the observed particle region, enabling the network to primarily recover the invisible region, ultimately outputting a complete particle non-modal mask.

[0102] 1. Prior branch of diffusion structure:

[0103] For the target particle to be completed, a local RGB region containing the particle is extracted from the original RGB image and input into a pre-trained Stable Diffusion model. Preferably, U-Net from Stable Diffusion v2.1 is used as the structural prior feature extraction network, and the parameters of this pre-trained diffusion network are kept frozen during the training of the non-modal completion network. After the input RGB region is processed by the latent representation corresponding to Stable Diffusion, multi-scale intermediate features from different encoding stages of U-Net are extracted at a selected diffusion time step.

[0104] In a specific setting: This parameter represents a specific preferred time step, but the technical concept of this invention is not limited to a single fixed time step. Features extracted from different coding layers are mapped to form a priori representation of the diffusion structure: The diffusion structure prior is used to characterize information related to the overall shape organization, contour continuity trend, and spatial structure of particles, thereby providing a basis for structural reconstruction of particle regions that are completely invisible in the current image.

[0105] It should be noted that the purpose of this branch is not to generate new RGB images of particles, but to obtain the structural representation of the pre-trained diffusion network. Therefore, there is no need to perform the complete stepwise backdiffusion image generation process.

[0106] 2. RGB-D geometry guiding branch:

[0107] While diffusion structure priors can provide overall structural information, they cannot directly represent the actual spatial geometry of the current industrial particle. Therefore, this invention establishes an independent RGB-D geometry guidance branch to encode the geometric information actually observed from the current target particle. This branch takes the incomplete visibility mask of the target particle and the corresponding local depth map as input and combines them along the channel direction.

[0108] The visibility mask describes the area currently observed by the particle, the location of the particle's true visible boundary, and the spatial relationship between the invisible and visible areas. The depth map describes the depth variation on the particle surface, the spatial height of the particle in the stacked scene, and the relative depth relationship between the target particle and surrounding particles.

[0109] The aforementioned dual-channel information is input into a convolutional encoder with residual connections to extract local geometric features at different spatial resolutions layer by layer, and the obtained RGB-D geometric representations are uniformly denoted as... Therefore, the diffusion structure is a priori. With RGB-D geometric representation They serve different functions: the former provides the overall shape and contour continuity information needed for the recovery of hidden regions, while the latter is used to ensure that the recovery result is consistent with the actual visible geometry of the current particles.

[0110] 3. Multi-scale cross-modal post-fusion:

[0111] After feature extraction is completed in both branches, multi-scale post-fusion is performed in the decoding stage. At the corresponding spatial resolution, the diffusion structure prior features and RGB-D geometric features are concatenated along the channel direction and then fed into the subsequent decoding network.

[0112] By employing post-fusion instead of directly combining the two types of information at the input stage, the diffusion branch can first preserve its pre-trained latent structural representation, while the RGB-D branch independently extracts the actual geometric constraints of the current particle. Then, during the reconstruction process, the correspondence between the two types of information is established layer by layer. The fused features are then reconstructed into the particle mask space through convolution and progressive upsampling, and finally, the initial non-modal prediction probability map is output via the Sigmoid function. The prediction result includes both the currently visible particle regions and the hidden regions inferred by the network.

[0113] 4. Visible mask preservation constraints:

[0114] Since the particle region in the visible mask has already been observed by the sensor, the present invention further sets a physical consistency maintenance constraint based on the visible mask, so that non-modal completion is only responsible for restoring the invisible part, without arbitrarily modifying the observed particle region.

[0115] Let the incomplete visibility mask of the input be... The initial output of the network is The final constrained nonmodal prediction result is ,but:

[0116] ;

[0117] in, This is the initial predicted probability map. This is an incomplete visibility mask for the particles. The output is the result after maintaining constraints through the visible region. This represents element-wise product.

[0118] when When the corresponding position belongs to the actually observed particle region, the output is directly maintained as the particle region; when At that time, the corresponding location belongs to an undetermined region, and whether it belongs to a hidden particle is determined by the network prediction value.

[0119] Therefore, the final complete particle mask consists of both the "truly visible part" and the "invisible part recovered by the model". After thresholding, the complete non-modal mask of the corresponding occluded particle can be obtained.

[0120] 5. Training of nonmodal completion networks:

[0121] The DSDA-Net was trained under supervision using the aforementioned multimodal occlusion samples.

[0122] For each training sample, the synthesized RGB region, the corresponding depth map, and the incomplete visible mask are used as network inputs, along with the Base-Up complete mask retained before synthesis. As a supervised objective, the training loss of the two-branch nonmodal completion network consists of binary cross-entropy loss and Dice loss:

[0123] ;

[0124] in, For binary cross-entropy loss, For Dice's loss, and These are the weights corresponding to the two loss terms.

[0125] Through the above training, the network establishes a mapping relationship between partially visible RGB-D observations and complete particle masks.

[0126] V. Fusion of complete particle instances:

[0127] For unoccluded particles (UP) identified during the perception phase, their currently visible mask is directly retained as the complete particle mask. For occluded particles (OP), the non-modal completion result output by DSDA-Net is used as the complete particle mask.

[0128] To avoid instance mixing after multiple mutually occluded particles are remapped to the same 2D binary image, this invention employs a channel-level particle instance organization method. Each particle instance occupies an independent channel, and all particle channels are then combined into a unified particle instance tensor. This fusion method preserves the complete mask of each particle while maintaining the instance independence and original stacking relationships between different particles. The resulting complete set of particle instances is used for subsequent physical granularity calculations.

[0129] VI. Granularity distribution estimation based on depth perspective correction and field of view edge compensation:

[0130] Nonmodal completion recovers the complete projected geometry of particles in image space, but final industrial particle size measurement requires obtaining the actual physical size. Therefore, this invention further utilizes depth information acquired by an RGB-D sensor to establish a dynamic scale transformation relationship at the particle level, as follows:

[0131] 1. Depth-related perspective scale correction:

[0132] For the For each complete particle instance, extract the effective depth value within the particle mask region, and use the median as the representative depth of the particle. Using the depth of the particles themselves, rather than a uniform imaging distance across the entire image, can reflect the scale differences between particles with different stacking heights. The physical scale conversion factor of a particle is expressed as:

[0133] ;

[0134] in, For the first The scale transformation coefficient of each particle. This represents the representative imaging depth of the particle. The focal length parameters obtained from camera calibration These are calibration parameters related to the sensor pixel scale.

[0135] By determining each particle individually This enables dynamic scale correction that varies with particle depth.

[0136] 2. Calculation of equivalent diameter of intact particles:

[0137] For particles not truncated by image boundaries, their projected pixel area is calculated based on the non-modal complete mask. Subsequently, the Heywood equivalent diameter was used to represent the uniform size of the irregular particles. Its actual physical equivalent diameter is expressed as:

[0138] ;

[0139] in, The pixel area corresponding to the complete mask of the particle. This represents the dynamic scale transformation coefficient of the current particle. The physical Heywood equivalent diameter of the particle.

[0140] Therefore, the complete image spatial geometry obtained through nonmodal completion can be directly converted into the actual particle size.

[0141] 3. Field of view edge truncation particle size compensation:

[0142] For particles that intersect with the field of view boundaries of the image, part of their outlines lie outside the camera's field of view. Even if the occluded area inside the particle is reconstructed, the complete projected area outside the image cannot be directly obtained. Therefore, this invention first determines whether the particle mask is in contact with the image boundary.

[0143] For particles that do not touch the image boundary, the Heywood equivalent diameter is calculated as described above. For particles that intersect the image boundary, they are identified as field-of-view truncated particles, and the effective particle boundary points still within the image are extracted. The equivalent radius corresponding to the particle is estimated by least-squares circle fitting. Let the fitted pixel equivalent radius be... The actual physical equivalent diameter after compensation is:

[0144] ;

[0145] in, The pixel equivalent radius obtained by fitting edge particles. This is the scale transformation factor for the particle. The equivalent diameter of the particle after field-of-view truncation compensation.

[0146] For cases where the effective boundary is too short or the local boundary is irregular, leading to unstable fitting, the fitting radius can be constrained by incorporating the size statistics of intact internal particles in the current field of view, thereby reducing the impact of abnormal boundaries on the particle size estimation results. This step is used to estimate the equivalent particle size corresponding to the missing region outside the field of view, rather than assuming that the real and complex particle contours outside the field of view can be recovered.

[0147] 4. Particle size distribution calculation:

[0148] Based on the above steps, the physical equivalent diameters of all effective particles in the current field of view are obtained. These are then classified and statistically analyzed according to preset particle size ranges to obtain a quantitative particle size distribution. For the same type of industrial material, when the differences in material density and shape factor between different particles are small, a mass-equivalent weighting can be performed using the cube of the particle size to convert the quantitative particle information into a mass-equivalent particle size distribution. For the first... Each particle size range, its equivalent mass percentage Represented as:

[0149] ;

[0150] in, For the first The number of particles in each particle size range This refers to the total number of particles involved in particle size statistics. The first [value] obtained after depth scale correction and necessary field-of-view edge compensation The physical equivalent diameter of each particle. This invention proposes an RGB-D nonmodal completion and particle size distribution estimation method for densely occluded industrial material particles. This method can be applied to industrial particle scenarios such as sintered ore, coke, pellets, and ores, which have irregular shapes and are prone to adhesion and occlusion. The method completes non-contact particle size distribution measurement of industrial material particles through four stages: data acquisition and preparation, training sample construction and model training, on-site inference and particle size calculation, and industrial deployment and operation maintenance. The specific implementation process is as follows.

[0151] 1. Data Collection and Preparation Stage:

[0152] (1) Industrial particle RGB-D data acquisition:

[0153] An RGB-D vision acquisition device is installed above the industrial material conveyor belt or particle detection area to ensure that the camera's field of view covers the main material conveying area and that the particles have sufficient spatial resolution in the image. The RGB image is used to acquire the color, texture, and two-dimensional contour information of the particles, while the depth map is used to acquire the particle surface height, spatial distance, and the layering information between particles.

[0154] The RGB-D particle data acquisition system used in this embodiment is as follows: Figure 1 As shown, an industrial camera and a lidar are mounted above the industrial material conveyor belt via a calibration bracket. The industrial camera is used to acquire RGB images of particles within the detection area, while the lidar is used to acquire depth data for the corresponding area. The effective observation ranges of the two types of sensors overlap within the conveyor belt's detection area to ensure that the RGB and depth information correspond to the same particle region. After industrial material particles on the conveyor belt continuously pass through the detection area, the RGB-D data required for subsequent particle sensing, non-modal completion, and particle size distribution estimation can be obtained.

[0155] During data acquisition, RGB images and depth maps should be acquired synchronously and spatially registered to ensure that pixel values ​​in the RGB image correspond to their respective positions in the depth map. For continuous conveyor belt operation scenarios, RGB-D images can be acquired continuously at a fixed sampling frequency, with the sampling frequency adjusted based on the on-site conveyor speed, particle size range, and computing power. In industrial environments with dust, vibration, or varying lighting conditions, the camera mounting structure should be stable, and appropriate industrial supplementary lighting and dustproof measures should be implemented to minimize the impact of the external environment on the quality of RGB images and depth data.

[0156] (2) Data filtering and dataset partitioning:

[0157] The acquired RGB-D data is subjected to quality screening to remove invalid samples that are severely blurred, overexposed, underexposed, have large-scale depth loss, have obvious misalignment between RGB and depth, or have grain areas that cannot be effectively identified, and retain valid data that meet the requirements for subsequent processing.

[0158] The effective data is divided into training, validation, and test sets. The training set is used for training the non-modal completion model, the validation set is used for model parameter selection and training process evaluation, and the test set is used for final performance verification. The specific division ratio can be set according to the actual data collection scale, ensuring that different data subsets contain samples with different particle sizes, occlusion levels, and stacking states.

[0159] For training data, appropriate data augmentation techniques such as random rotation, horizontal flipping, and scale changes can be applied to increase particle pose and appearance variations, thereby improving the model's adaptability to different industrial scenarios. Data augmentation should maintain spatial consistency between the RGB image, depth map, and corresponding mask to avoid cross-modal positional misalignment.

[0160] (3) RGB-D data preprocessing:

[0161] The RGB image is resized and normalized, and invalid and outlier values ​​in the depth map are processed as necessary, while maintaining the original depth structure and grain boundary relationships. The RGB image and depth map are unified to the same spatial coordinate system before entering subsequent processing.

[0162] The initial visible regions of industrial material particles are obtained using existing particle segmentation methods. For the complete particle samples required for subsequent training, particle instances with complete outlines and no occlusion are selected from actual images as the base data for automatically generating non-modal training samples.

[0163] 2. Training Sample Construction and Model Training Phase:

[0164] (1) Automatic construction of occluded training samples:

[0165] The specific process of generating occluded training samples and non-modal supervised annotations is as follows: Figure 6 As shown. First, base particles and occlusion particles are sampled from the complete set of visible particles, such as... Figure 6 As shown in (a); then, a simulated visual occlusion is formed through spatial transformation, such as... Figure 6 As shown in (b), depth data alignment and fusion are completed according to the particle front-to-back relationship, as follows. Figure 6 As shown in (c); based on this, extract the corresponding tags of the basic particles, such as Figure 6 As shown in (d), modal visibility masks after occlusion and complete non-modal supervision annotations before occlusion are formed, respectively, as follows. Figure 6 (e) and Figure 6 As shown in (f).

[0166] Two particle instances are selected from the complete set of visible particles, serving as the base particle and the occluding particle, respectively. Spatial transformations such as rotation, scaling, and translation are performed on the occluding particle to create different positions and degrees of overlap with the base particle, thereby simulating the occlusion phenomenon between particles in an industrial setting.

[0167] After completing the two-dimensional spatial overlay, the occluded particles are locally adjusted according to the depth information corresponding to the two particles, so that the occluded particles remain in front of the base particles in the overlapping area, and the occlusion relationship in the RGB image is consistent with the spatial front-back relationship in the depth map.

[0168] After completing the simultaneous fusion of RGB, depth, and mask, the complete mask before occlusion of the basic particles is used as a non-modal supervision annotation, and the part that is still visible after occlusion is used as an incomplete visible mask. This automatically generates RGB-D occlusion samples for network training. By changing the particle combination, occlusion position, rotation angle, and scale, training samples with different occlusion patterns can be constructed, improving the model's adaptability to real-world densely stacked scenes.

[0169] (2) Model training environment configuration:

[0170] Model training can be performed on a workstation or server with GPU acceleration capabilities. The hardware configuration should meet the memory and computational requirements for joint training of Stable Diffusion feature extraction, RGB-D feature encoding, and non-modal completion networks.

[0171] The software environment can be built based on mainstream deep learning frameworks, ensuring compatibility with deep learning libraries, GPU drivers, and CUDA runtime environments. Model weights and training state are saved periodically during training to allow for resumption after training interruptions.

[0172] (3) Training the DSDA-Net model:

[0173] The DSDA-Net was trained using the automatically constructed RGB-D occlusion samples described above. During training, the input included the RGB local image, depth map, and incomplete visible mask corresponding to the target particle, while the supervision target was the corresponding complete particle non-modal mask.

[0174] The diffusion structure prior branch uses a pre-trained Stable Diffusion v2.1 model with its U-Net parameters frozen to extract multi-scale structural features for particle hidden structure recovery. The RGB-D geometry branch uses visible particle masks and depth information to extract the local geometric constraints of the current target particle. The two types of features are fused in the decoding stage to output the complete particle prediction result.

[0175] In one specific setup, an intermediate diffusion time step of t=181 can be selected for structural feature extraction; this time step is a preferred setting, but other intermediate diffusion time steps can also be selected based on training data and validation results. During training, a validation set is used to monitor the model's recovery effect on intact particle regions and hidden regions, and the weights of the best-performing models are saved based on the validation results. The specific number of training rounds, batch size, learning rate, and optimizer parameters can be set according to the data scale, computing platform, and validation results, without affecting the overall implementation process of this invention.

[0176] 3. On-site reasoning and granularity calculation stage:

[0177] (1) Particle instance separation and occlusion recognition:

[0178] After the model training is completed, the trained nonmodal completion model is deployed on the field computing equipment. For industrial particle RGB-D images acquired in real time or periodically, the visible region of the particles is first obtained, and the adhering particles are separated into instances using two-dimensional geometric information and depth information.

[0179] The separation process of adhering particles is as follows: Figure 4 As shown. First, based on the particle regions and their extreme points in the original RGB image, candidate locations corresponding to different particle instances within the adhered particle regions are determined. Figure 4 (a) shows the original RGB image and the extreme points of the particle region; at the same time, the corresponding three-dimensional depth information is used to characterize the changes in particle surface height and the spatial hierarchy between adjacent particles, such as... Figure 4 As shown in (b). In obtaining Figure 4 (c) shows the initial adherent particle mask. A topological response is constructed by combining the geometric and depth information of the particle region, and seed points for instance partitioning are generated accordingly. Figure 4 As shown in (d); subsequently, the seed point is used as a separation marker to divide the region of adherent particles into instances, resulting in... Figure 4 (e) shows the mask for the independent particle instance.

[0180] For each independent particle instance obtained, its occlusion status is determined based on the depth variation relationship inside and outside the particle boundary. Particles without obvious occlusion boundaries are directly retained as unoccluded particles; particles with occlusion boundaries are treated as occluded particles and enter the subsequent non-modal completion process.

[0181] The actual process of identifying particle occlusion is as follows: Figure 5 As shown. Figure 5 (a) The separated particle instance mask is superimposed on the corresponding RGB image to determine the particle to be judged and its boundary position; Figure 5 (b) provides depth information for the same region, and the spatial hierarchy between different stacked particles can be observed through changes in depth; Figure 5 (c) The depth gradient direction at the particle boundary is further shown. By comparing the depth changes on both sides of the boundary, it is determined whether the boundary is covered by other particles. Based on the above determination results, each particle instance is divided into unoccluded particles (UP) and occluded particles (OP). The classification results are as follows: Figure 5 As shown in (d), unoccluded particles can directly retain their complete visible structure, while occluded particles are filtered out and enter the subsequent DSDA-Net nonmodal completion process.

[0182] (2) Nonmodal completion of occluded particles:

[0183] For identified occluded particles, the corresponding RGB local image, depth map, and visibility mask are cropped and input into the trained DSDA-Net for non-modal completion. The network utilizes the structural priors provided by the diffusion model and the actual geometric information provided by RGB-D observations to recover the invisible particle regions, and preserves the actually observed particle regions through visibility mask preservation constraints.

[0184] Unoccluded particles do not require non-modal completion; they are directly recombinated with the completed occluded particles. Different particles maintain their independent instance form, thus obtaining a complete set of particle instances in the current field of view.

[0185] (3) Calculation of physical particle size:

[0186] For each complete particle instance, a representative depth value is obtained from the corresponding depth region, and the pixel-physical size conversion relationship corresponding to the particle is determined in combination with the camera calibration parameters to compensate for the perspective scale difference caused by different particle stacking heights.

[0187] For particles that are completely within the field of view, their physical equivalent particle size is calculated based on the area of ​​the complete mask. For truncated particles that intersect with the image edge, the equivalent size is compensated using the boundary information that is still valid within the field of view, reducing the size underestimation caused by the particle part being located outside the field of view.

[0188] like Figure 8 As shown, this embodiment employs a differentiated particle size calculation method for different particles based on whether or not the particles intersect with the image field of view boundary. Figure 8 (a) First, identify edge particles that contact the image boundary and internal particles that are completely within the field of view in the complete particle segmentation result; for internal particles, such as Figure 8 As shown in (b), the physically equivalent particle size is calculated using the projected area of ​​its complete non-modal mask and the corresponding depth scale transformation relationship; for particles truncated by the field of view boundary, such as Figure 8 As shown in (c), since the complete projected area cannot be obtained directly, the visible boundary within the effective field of view is extracted, and the equivalent particle size is estimated by least-squares circle fitting. Then, the compensated physical equivalent particle size is obtained by combining the depth scale transformation relationship of the corresponding particles. By distinguishing between internal intact particles and truncated particles at the edge of the field of view, the particle size of edge particles can be underestimated due to using the same area calculation method for the two types of particles. After completing the particle size calculation for all effective particles, the particle quantity distribution is statistically analyzed according to the particle size interval set in the production site. For applications requiring mass-type particle size distribution, mass-equivalent weighting can be performed using particle size cube under the condition that the density and shape factor of similar materials are relatively small, thereby obtaining the corresponding mass-type particle size distribution.

[0189] (4) Output of test results:

[0190] Based on the on-site production needs, the final obtained average particle size, proportion of different particle size ranges, quantity-based particle size distribution, or quality-equivalent particle size distribution will be output to the host computer, industrial monitoring platform, or production process database.

[0191] For continuous delivery scenarios, the detection results of multiple frames can be summarized according to a preset time window. For example, the particle size distribution within a certain time interval can be statistically analyzed to reduce the impact of the limited number of particles in a single frame or the instantaneous change in the accumulation state on the statistical results.

[0192] 4. Industrial Deployment and Operation / Maintenance Phase:

[0193] (1) Installation and calibration of vision equipment:

[0194] RGB-D vision devices should be installed in a relatively stable position above a conveyor belt or particle flow, ensuring that particles within the detection area can completely enter the camera's effective field of view. After installation, spatial calibration of the RGB image and depth map should be performed, along with geometric calibration between pixel size and actual physical size.

[0195] After equipment maintenance, camera relocation, or significant mechanical vibration, the registration status of RGB and depth data and camera calibration parameters should be rechecked to avoid granularity conversion errors caused by changes in installation location.

[0196] (2) Industrial environment adaptation:

[0197] For scenarios with high dust levels, such as sintered ore and coke, a protective cover can be installed on the outside of the vision equipment, and an air purging device can be configured according to the site conditions to reduce image blurring and depth measurement abnormalities caused by dust adhering to the lens.

[0198] For areas with significant ambient light variations, a stable industrial supplemental lighting device can be installed, and the lighting angle adjusted to reduce the impact of grainy shadows, strong reflections, and light fluctuations on RGB image quality. During equipment installation, vibrations from the conveyor equipment should also be minimized to ensure the stability of the field of view and camera calibration during long-term data acquisition.

[0199] (3) Computing equipment and model deployment:

[0200] The training phase can be completed using a dedicated GPU server, while the inference phase can be performed using an industrial computer or an edge computing device with GPU computing capabilities, depending on the actual processing speed and system cost. During deployment, a complete processing program is composed of modules for granular segmentation, instance separation, occlusion recognition, DSDA-Net nonmodal completion, and granular statistics, and a continuous data interface is established with the RGB-D data acquisition module.

[0201] Since nonmodal completion is only performed on particle instances that are determined to be occluded, unoccluded particles directly enter the granularity calculation, which can reduce unnecessary model inference and improve the efficiency of on-site computing resource utilization.

[0202] (4) Material type changes and model adaptation:

[0203] When the object of detection is changed from one type of industrial material to another material that has significant differences in appearance, shape or surface structure, the RGB-D data of the material can be re-acquired, and complete visible particles can be screened from them to automatically construct new occlusion training samples according to the aforementioned method.

[0204] Based on the differences between the new material and the original training material, the nonmodal completion module can be adapted or retrained on the basis of the original model without manually drawing a large number of hidden particle boundaries one by one, thereby reducing the amount of data preparation work when changing the type of industrial particles.

[0205] (5) Maintenance of running results:

[0206] During system operation, particle size, particle size distribution, and representative RGB-D images can be periodically saved for subsequent production quality analysis and equipment status checks. When persistent anomalies are observed in the detection results, factors such as camera lens contamination, missing depth data, RGB-D spatial misalignment, changes in ambient lighting, and changes in material type should be prioritized for investigation. Based on the actual situation, the equipment should be recalibrated or the model adjusted.

[0207] Through the above implementation methods, this invention can form a complete application process from industrial particle RGB-D data acquisition, training sample generation, and non-modal model training, to on-site particle occlusion recovery, physical particle size calculation, and particle size distribution output, enabling the proposed technical solution to be deployed in actual industrial material conveying and particle detection scenarios. Experimental results show that the non-modal completion model of the complete RGB-D version of this invention achieves 85.61% Amo-mIoU and 70.03% Inv-mIoU. Among them, in the Inv-mIoU index, which mainly reflects the recovery ability of invisible areas, it is improved by 5.91 percentage points compared with the higher-performing C2F-Seg of the compared methods (64.12%), indicating that this invention has better recovery ability for occluded and hidden areas. Compared with DSDA-Net that only uses RGB information, after introducing depth geometric information, Amo-mIoU increased from 82.48% to 85.61%, and Inv-mIoU increased from 64.41% to 70.03%, indicating that RGB-D geometric constraints can further improve the hidden area recovery results. Experimental results show that after adopting the complete technical solution, the RMSE of particle size estimation is 6.84 mm, and the MAE is 5.31 mm. Compared with the benchmark process of directly measuring visual particle size based on SAM segmentation results, the RMSE is reduced from 14.32 mm to 6.84 mm, a decrease of approximately 52.23%, and the MAE is reduced from 11.85 mm to 5.31 mm, a decrease of approximately 55.19%. Furthermore, by gradually introducing non-modal completion, perspective correction, and edge compensation on the basis of already good particle segmentation, the RMSE is further reduced from 11.65 mm to 6.84 mm, indicating that the complete particle recovery and geometric calibration technology adopted in this invention can improve the subsequent physical particle size and particle size distribution estimation results.

Claims

1. A method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles, characterized in that, Includes the following steps: Step 1: RGB-D Particle Data Acquisition and Visible Region Extraction: Acquire RGB and depth images that are synchronously collected and spatially registered in the industrial material particle scene, so that the pixel positions in the RGB image correspond to the depth measurement positions in the depth image; and obtain the visible region mask of the industrial material particles based on the RGB image and / or depth image. Step 2: Separation of Adhesive Particle Instances Based on Geometric and Depth Information: For an adhesive connected region formed by multiple particles in the image space, a geometric potential field is constructed based on the distance from the pixels inside the connected region to the boundary, and a depth potential field is constructed based on the depth data corresponding to the connected region; the geometric potential field and the depth potential field are fused to construct a fused topographic map, the particle instance marker positions are determined based on the fused topographic map, and the adhesive connected region is segmented using the particle instance marker positions to obtain mutually independent particle instance masks; Step 3: Occlusion state recognition based on particle boundary depth relationship: For each particle instance that has been separated, local depth information is obtained on the inside and outside of the particle boundary respectively. Based on the depth difference inside and outside the particle boundary, it is determined whether the particle has an occlusion boundary covered by other particles. When there are other particle surfaces on the outside of the particle boundary that are closer to the RGB-D vision acquisition device than the inside of the particle, and the depth difference exceeds the preset threshold used to suppress the influence of depth measurement noise, the corresponding boundary is determined to be an occlusion boundary. Based on whether there is an effective occlusion boundary, particle instances are divided into unoccluded particles and occluded particles; Step 4: Automatic generation of multimodal occlusion training samples and nonmodal supervised annotations: Select at least two complete visible particles from the set of unoccluded particles, one as the base particle and the other as the occluded particle; perform at least one spatial transformation on the occluded particle, including rotation, scaling and translation, so that the transformed occluded particle and the base particle have effective spatial overlap. Based on the depth information corresponding to the base particles and the occluded particles, the transformed occluded particles are depth aligned so that the occluded particles have a depth relationship in front of the base particles in the overlapping area. Then, the RGB information, depth information and particle mask of the two particles are combined synchronously to form RGB-D occlusion training samples with consistent front and back occlusion relationships. The complete mask before the basic particle is occluded is used as the non-modal supervision annotation of the training sample, and the remaining visible area after the basic particle is occluded is used as the incomplete visible mask. Step 5, Nonmodal Completion Network Training: Train a bi-branch nonmodal completion network using the RGB-D occlusion training samples obtained in Step 4. The bi-branch nonmodal completion network includes a diffusion structure prior extraction branch and an RGB-D geometry guidance branch. Step 6, Non-modal Completion of Occluded Particles: The RGB local image corresponding to the occluded particle to be completed is input into the diffusion structure prior extraction branch. Intermediate latent features at different levels are extracted as diffusion structure priors through a pre-trained diffusion model. The incomplete visible mask of the particle to be completed and its corresponding local depth map are input into the RGB-D geometry guidance branch. RGB-D geometric features are obtained through convolutional encoding. The diffusion structure prior and RGB-D geometric features are fused at corresponding feature scales, and the initial non-modal prediction result of the occluded particle is obtained through decoding. Step 7, Visible Area Preservation Constraint: The initial non-modal prediction result is constrained by the incomplete visible mask of the particle to be completed, so that the particle area already observed by RGB-D visual data remains unchanged, and a complete non-modal particle mask containing the visible area of ​​the particle and the recovered invisible area is obtained. Step 8: Complete Particle Instance Fusion: For unoccluded particles obtained through occlusion state identification, their original complete visible masks are retained; for occluded particles, complete non-modal masks are obtained using a non-modal completion network. The unoccluded particles and the occluded particles that have completed nonmodal completion are reorganized according to their spatial positions in the original image to obtain a complete set of particle instances. Step 9, perspective scale correction: For each particle in the complete set of particle instances, extract effective depth data from its corresponding depth region to determine the representative imaging depth, and combine the representative imaging depth and camera calibration parameters to determine the scale transformation relationship of the corresponding particle from pixel size to physical size. Step 10, Calculation of Particle Physical Size and Compensation for Field of View Edge: Determine whether the particles intersect with the image field of view boundary. For particles that do not intersect with the field of view boundary, calculate the physical equivalent particle size based on the complete particle mask and the corresponding scale transformation relationship. For truncated particles that intersect with the field of view boundary, estimate the equivalent size based on the boundary within the effective field of view, and obtain the compensated physical equivalent particle size by combining the corresponding scale transformation relationship. Step 11, Particle size distribution calculation: Based on the physical equivalent particle size of each effective particle, the quantity is statistically analyzed according to the preset particle size range. Under the condition that the differences in particle material density and shape factor of similar industrial materials can be approximately ignored, the mass equivalent weighting is performed based on the cube of the physical equivalent particle size to obtain the mass equivalent particle size distribution of the industrial material particles.

2. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step two, the geometric potential field is obtained by performing an Euclidean distance transformation on the binary mask and establishing a normalized geometric potential field based on the distance from each pixel within the connected region to the background boundary; the depth potential field is obtained by extracting the depth region corresponding to the connected region from the depth map and performing smoothing and normalization processing; the fused terrain map is constructed as follows: ; in, It is a geometric potential field; For deep potential field, and These are the weights corresponding to the two types of potential fields. Represents element-wise product. It is a matrix of all ones with the same size as the input mask.

3. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step three, the pixel positions on the particle boundary are selected along the boundary normal direction, with the inner and outer neighborhoods of the particle being selected respectively. The local depth gradient at the boundary position is defined as: ; in, The average depth of the neighborhood inside the boundary. The average depth of the neighborhood outside the boundary; when the region of a continuous boundary satisfies: At that time, among them, The boundary is determined as an occlusion boundary by a threshold used to suppress the influence of noise in depth sensor measurements.

4. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, The depth alignment in step four includes: calculating the depth offset based on the depth statistics of the base particle and the occluded particle in the corresponding area, and introducing a positive depth offset to keep the occluded particle in front of the base particle in the overlapping area.

5. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, The diffusion structure prior extraction branch in step five uses a pre-trained StableDiffusion model and extracts its U-Net multi-scale intermediate latent features at different encoding stages as structure priors; the U-Net parameters of the pre-trained StableDiffusion model are frozen during the training of the non-modal completion network.

6. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step seven, the visibility mask preservation constraint is applied using the initial non-modal prediction results obtained in step six in the following manner: ; in, This is the initial predicted probability map. This is an incomplete visibility mask for the particles. The output after maintaining constraints through the visible region. This represents element-wise product.

7. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step five, the training loss of the bi-branch nonmodal completion network consists of binary cross-entropy loss and Dice loss: ; in, For binary cross-entropy loss, For Dice's loss, and These are the weights corresponding to the two loss terms.

8. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step nine The physical scale conversion factor of each particle is determined as follows: ; in, For the first The scale transformation coefficient of each particle. This represents the representative imaging depth of the particle. The focal length parameters obtained from camera calibration These are calibration parameters related to the sensor pixel scale.

9. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, In step ten, for truncated particles that intersect with the image field of view boundary, the effective particle boundary points that are still inside the image are extracted, and the equivalent radius corresponding to the particle is estimated by least squares circle fitting. The physical equivalent diameter after truncation compensation is calculated as follows: ; in, The pixel equivalent radius obtained by fitting edge particles. This is the scale transformation factor for the particle. The equivalent diameter of the particle after field-of-view truncation compensation.

10. The method for RGB-D nonmodal completion and particle size distribution estimation of densely occluded industrial material particles according to claim 1, characterized in that, The particle size distribution mentioned in step eleven includes quantitative particle size distribution and mass-equivalent particle size distribution; for mass-equivalent particle size distribution, the first step is calculated as follows: Equivalent mass percentage of each particle size range: ; in, For the first The number of particles in each particle size range This refers to the total number of particles involved in particle size statistics. The first [value] obtained after depth scale correction and necessary field-of-view edge compensation The physical equivalent diameter of each particle For the first to participate in granularity statistics The physical equivalent diameter of each particle is obtained after perspective scale correction and necessary field-of-view edge compensation.