Solid waste particle intelligent identification method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of visual detection of solid waste particles, in particular to an intelligent identification method and system for solid waste particles based on multi-modal fusion, and the method comprises the steps: firstly, carrying out the particle segmentation and particle size statistics of a solid waste infrared image, and calculating a particle adhesion evaluation value; dividing the solid wastes into low, medium and high adhesion grades; then, for different adhesion levels, channel weighted fusion guided by infrared images, or frequency domain fusion guided by RGB images, or instance level alignment fusion based on three-dimensional point cloud space anchor points is adopted to obtain multi-modal fusion features; and finally, the infrared image and the multi-modal fusion feature are mapped to a three-dimensional point cloud model to construct a three-dimensional solid waste particle model, accurate geometric dimensions and material features are obtained at the particle instance level, and the accuracy of solid waste particle type recognition and particle size measurement and the overall recognition efficiency are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of visual inspection technology for solid waste particles, specifically to a method and system for intelligent identification of solid waste particles based on multimodal fusion. Background Technology

[0002] With the advancement of "zero-waste city" construction in my country, the refined and resource-oriented treatment of solid waste has become a crucial link. Traditional solid waste particle classification often uses mechanical sieving methods. For irregular particles such as construction solid waste, traditional sieving is time-consuming and labor-intensive, and cannot obtain particle morphological parameters (such as roundness and angularity), making it difficult to meet the needs of large-scale waste resource utilization. Chemical analysis methods can improve efficiency; for example, graphite furnace atomic absorption spectrophotometry can be used to determine the content of heavy metals (such as lead and cadmium) in solid waste. However, this method only provides chemical composition information and cannot obtain the physical morphological characteristics of particles. Moreover, the pretreatment is complex and costly. Image processing technology can effectively solve the efficiency and cost problems. However, solid waste particles are characterized by their variety, similar colors, and tendency to stick together, which brings great difficulties to image processing, resulting in low accuracy in particle type identification. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for intelligent identification of solid waste particles based on multimodal fusion.

[0004] The technical solution of this invention is as follows: A method for intelligent identification of solid waste particles based on multimodal fusion includes the following operations: S1: Acquire infrared images of solid waste, perform particle segmentation and particle size statistics to obtain solid waste particle size information; based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, calculate the particle adhesion assessment value; according to the particle adhesion assessment value, classify the adhesion attribute into low adhesion level, medium adhesion level, and high adhesion level, and execute S2, S3, and S4 respectively; the larger the particle adhesion assessment value, the higher the adhesion level. S2: Using a channel-weighted fusion method guided by infrared images, the solid waste regions of the infrared images and RGB images are fused to obtain a multimodal fusion feature map, and then S5 is executed; S3: Using a frequency domain fusion method guided by RGB images, the solid waste region of the infrared image and the RGB image are fused to obtain a multimodal fusion feature map, and then S5 is executed; S4: Using the particle center of the 3D point cloud model of solid waste as the spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and then execute S5. S5: Map infrared images and multimodal fusion feature maps onto a 3D point cloud model of solid waste to construct a 3D solid waste particle model; based on the 3D solid waste particle model, identify the particle type and size of each individual particle in the solid waste, thus forming the particle identification result of solid waste.

[0005] The specific operations of S2 are as follows: The solid waste region of the infrared image is divided into local windows; the local gradient energy of the infrared temperature value is calculated within each window and normalized to obtain an infrared thermal response instability map; the infrared thermal response instability map is then decoupled and smoothed to obtain an infrared component stability guide map; global statistics are performed on the infrared component stability guide map to obtain an infrared stability description vector; based on the infrared stability description vector, RGB channel weights and infrared channel weights are generated respectively; channel features are extracted from the solid waste region of both the RGB and infrared images to obtain RGB channel feature maps and infrared channel feature maps; based on the RGB channel weights and infrared channel weights, the RGB channel feature maps and RGB channel weights are weighted and summed to obtain a multimodal fusion feature map.

[0006] The method for obtaining solid waste regions from infrared images is as follows: The infrared image is scale-smoothed to obtain the background temperature field; the difference between the infrared image and the background temperature field is calculated to obtain an infrared foreground enhancement map. After normalization, the probability value of each pixel belonging to a solid waste region is calculated based on the temperature amplitude and local temperature gradient. The probability values ​​of all pixels are combined to form an infrared foreground probability map; based on the histogram distribution of probability values ​​in the infrared foreground probability map, an adaptive foreground segmentation threshold is determined. Pixels with probability values ​​greater than the threshold are marked as solid waste regions, and the remaining pixels are marked as background regions, resulting in an initial solid waste region map; connected component analysis is performed on the initial solid waste region map to remove connected regions with areas smaller than a preset area threshold, and morphological closing operations are performed on the remaining regions to obtain the solid waste regions in the infrared image.

[0007] The specific operation of S3 is as follows: Construct a particle boundary consistency guidance map based on the solid waste region of the RGB image. After converting the solid waste region of the infrared image and the RGB image to the frequency domain, they are divided into low-frequency components, mid-frequency components and high-frequency components according to the frequency radius. Based on the particle boundary consistency guidance map, the RGB spectrum components and infrared spectrum components of different frequencies are weighted and modulated to obtain the low-frequency component modulation weight, mid-frequency component modulation weight and high-frequency component modulation weight of the RGB spectrum components and infrared spectrum components, respectively. At the corresponding frequency position, the infrared spectrum components of different frequencies are weighted and superimposed with the RGB spectrum components to obtain the frequency domain fusion feature. After inverse frequency transformation, a multimodal fusion feature map is obtained.

[0008] In S5, the point cloud in the three-dimensional solid waste particle model is initially clustered and corrected to obtain single particle three-dimensional instances; the geometric features, infrared features, and fusion attribute features of each three-dimensional particle instance are obtained, input into the pre-trained classification model, and the particle type and confidence score are output. Together with the particle size parameter of the corresponding single particle, the particle recognition result is formed; all particle recognition results form the particle recognition result of solid waste.

[0009] Obtain the image texture quality value of the RGB image; determine whether the image texture quality value is greater than the image texture quality value threshold; if it is not greater, perform image enhancement processing on the RGB image to obtain an RGB enhanced image, which is used to perform the operation in S2, S3, or S4.

[0010] The image enhancement process involves converting the RGB image to the frequency domain, adaptively weighting the different frequency components based on their energy distribution and angular consistency characteristics, and then converting the modulated spectrum back to the spatial domain to obtain the enhanced RGB image.

[0011] A multimodal fusion-based intelligent identification system for solid waste particles, used to implement the aforementioned multimodal fusion-based intelligent identification method for solid waste particles, includes: The adhesion level generation module acquires infrared images of solid waste, performs particle segmentation and particle size statistics to obtain solid waste particle size information; based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, it calculates particle adhesion assessment values; according to the particle adhesion assessment values, the adhesion attributes are divided into low adhesion level, medium adhesion level, and high adhesion level, corresponding to the execution of the first fusion module, the second fusion module, and the third fusion module, respectively; the higher the particle adhesion assessment value, the higher the adhesion level; The first fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a channel-weighted fusion method guided by the infrared image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The second fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a frequency domain fusion method guided by the RGB image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The third fusion module is used to use the particle center of the three-dimensional point cloud model of solid waste as a spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and execute the particle recognition result generation module. The particle identification result generation module is used to map infrared images and multimodal fusion feature maps onto a three-dimensional point cloud model of solid waste to construct a three-dimensional solid waste particle model. Based on the three-dimensional solid waste particle model, the particle type and particle size of each individual particle in the solid waste are identified, thus forming the particle identification result of the solid waste.

[0012] A multimodal fusion-based intelligent identification device for solid waste particles includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the aforementioned multimodal fusion-based intelligent identification method for solid waste particles.

[0013] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described intelligent identification method for solid waste particles based on multimodal fusion.

[0014] The beneficial effects of this invention are as follows: This invention provides a multimodal fusion-based intelligent identification method for solid waste particles. First, by segmenting and statistically analyzing infrared images of solid waste particles, the method calculates features such as the proportion of fine particles, particle size distribution concentration, and equivalent particle size to obtain particle adhesion assessment values. Based on these values, solid waste is classified into low, medium, and high adhesion levels. Then, for different adhesion levels, the method employs infrared image-guided channel-weighted fusion, RGB image-guided frequency domain fusion, or instance-level alignment fusion based on 3D point cloud spatial anchor points, respectively. This fully leverages the advantages of infrared images, RGB images, and 3D geometric information in their respective applicable scenarios to obtain more discriminative multimodal fusion features. Finally, the infrared images and multimodal fusion features are mapped onto a 3D point cloud model to construct a 3D solid waste particle model. This simultaneously obtains accurate geometric scale and material characteristics at the particle instance level, thereby significantly improving the accuracy and overall identification efficiency of solid waste particle type identification and particle size measurement while reducing invalid calculations and missegmentation. Attached Figure Description

[0015] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0016] In the attached diagram: Figure 1 This is a flowchart illustrating the method of this embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0018] This embodiment provides a method for intelligent identification of solid waste particles based on multimodal fusion. See [link to relevant documentation]. Figure 1 This includes the following operations: S1: Acquire infrared images of solid waste, perform particle segmentation and particle size statistics to obtain solid waste particle size information; based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, calculate the particle adhesion assessment value; according to the particle adhesion assessment value, classify the adhesion attribute into low adhesion level, medium adhesion level, and high adhesion level, and execute S2, S3, and S4 respectively; the larger the particle adhesion assessment value, the higher the adhesion level. S2: Using a channel-weighted fusion method guided by infrared images, the solid waste regions of the infrared images and RGB images are fused to obtain a multimodal fusion feature map, and then S5 is executed; S3: Using a frequency domain fusion method guided by RGB images, the solid waste region of the infrared image and the RGB image are fused to obtain a multimodal fusion feature map, and then S5 is executed; S4: Using the particle center of the 3D point cloud model of solid waste as the spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and then execute S5. S5: Map infrared images and multimodal fusion feature maps onto a 3D point cloud model of solid waste to construct a 3D solid waste particle model; based on the 3D solid waste particle model, identify the particle type and size of each individual particle in the solid waste, thus forming the particle identification result of solid waste.

[0019] The specific steps are detailed below.

[0020] S1: Acquire infrared images of solid waste, and obtain solid waste particle size information through particle segmentation and particle size statistics; based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, calculate the particle adhesion assessment value; according to the particle adhesion assessment value, classify the adhesion attribute into low adhesion level, medium adhesion level, and high adhesion level, and execute S2, S3, and S4 respectively; the larger the particle adhesion assessment value, the higher the adhesion level.

[0021] Based on infrared images of solid waste, the particle size information of solid waste particles is quickly analyzed, particle adhesion assessment value is calculated, and solid waste is divided into low, medium and high adhesion levels according to the assessment value. Different fusion strategies are implemented for solid waste particles with different adhesion characteristics to ensure that the best segmentation and recognition results can be achieved under different adhesion levels.

[0022] First, infrared images of solid waste are acquired. After particle segmentation and particle size statistics, the particle size information of solid waste is obtained. The specific steps are detailed below.

[0023] Step 1: Using the solid waste region in the infrared image as a regional constraint, construct an infrared particle-scale potential energy map to characterize the particle-scale distribution.

[0024] Specifically, the infrared image is segmented to separate the solid waste region from the background region, resulting in a solid waste region map. Multi-scale temperature gradient analysis is performed on the solid waste region map to extract infrared temperature gradient values ​​at different filtering scales, resulting in a multi-scale infrared thermal response feature map. After fusion processing, an infrared particle-scale potential energy map is obtained.

[0025] In the infrared particle-scale potential energy map, the region with a potential energy value greater than the first potential energy value threshold is used as the candidate region for particle-scale center and is used for the subsequent extraction of stable particle candidate points. The region with a potential energy value less than the second potential energy value threshold is used as the particle contact area or thermal transition area and is used for subsequent segmentation boundary constraints. The first potential energy value threshold is greater than the second potential energy value threshold.

[0026] The method for obtaining solid waste areas from infrared images is as follows.

[0027] The infrared image is scale-smoothed to obtain the background temperature field; the difference between the infrared image and the background temperature field is calculated to obtain the infrared foreground enhancement map, thereby significantly enhancing the thermal response contrast of the solid waste area.

[0028] The infrared foreground enhancement image is normalized. Based on the temperature amplitude and local temperature gradient, the probability value of each pixel belonging to the solid waste region is calculated. The probability values ​​of all pixels are combined to form the infrared foreground probability map. According to the histogram distribution of the probability values ​​in the infrared foreground probability map, the foreground segmentation threshold is adaptively determined. Pixels with probability values ​​greater than the probability value threshold are marked as solid waste regions, and the remaining pixels are marked as background regions, thus obtaining the initial solid waste region map.

[0029] Connectivity analysis is performed on the initial solid waste region map to remove connected regions with an area smaller than a preset area threshold. Morphological closure operations are then performed on the remaining regions to eliminate holes and breaks. Connected regions with morphological consistency that meet preset conditions are identified as the final solid waste regions, resulting in the solid waste region map of the infrared image.

[0030] Step 2: Extract local maxima from the infrared particle-scale potential energy map and perform scale stability analysis to obtain a candidate set of stable particles.

[0031] Specifically, local maxima are extracted from the infrared particle-scale potential energy map. The difference between the potential energy peak of each local maxima and the potential energy value of the nearest neighboring saddle point is obtained as its scale stability value. The scale stability value threshold is adaptively determined based on the statistical distribution of the scale stability values ​​of all local maxima. Local maxima with scale stability values ​​not less than the scale stability value threshold are selected as stable particle candidate points. All stable particle candidate points form a stable particle candidate set.

[0032] Step 3: Based on the solid waste region of the infrared image, construct a thermal boundary cost map. Under the constraint of the thermal boundary cost map, use guided segmentation propagation based on a stable particle candidate set to obtain the initial infrared particle segmentation map.

[0033] Thermal boundary cost maps are generated by normalizing infrared images, specifically the thermal texture feature maps of solid waste regions within infrared images, and then weighted and fused with infrared particle-scale potential energy maps. In the infrared thermal boundary cost map, regions with large temperature gradients and significant abrupt changes in thermal response have lower segmentation costs, while regions with gentler thermal responses, potentially belonging to the interior of particles, have higher segmentation costs.

[0034] The guided segmentation propagation operation based on a stable particle candidate set is as follows: using stable particle candidate points in the stable particle candidate set as segmentation seeds, the minimum propagation cost from each pixel to each stable particle-scale candidate point is calculated on the infrared thermal boundary cost map; pixels in the solid waste region of the infrared image are assigned to stable particle-scale candidate points with the minimum propagation cost, resulting in the initial infrared particle segmentation map. The minimum propagation cost is the minimum of the cumulative value of the thermal boundary cost of each pixel along the propagation path from the pixel to the corresponding stable particle-scale candidate point among all possible propagation paths.

[0035] Step 4: Perform adaptive correction based on particle size consistency on the initial infrared particle segmentation image to obtain the infrared particle segmentation image.

[0036] Specifically, the equivalent area difference, equivalent particle size difference, and infrared thermal response scale difference of each particle region in the initial infrared particle segmentation image are obtained, and the equivalent scale difference value is calculated by weighted summation. When the equivalent scale difference value of adjacent particle regions is less than the equivalent scale difference value threshold, and the thermal gradient intensity of the corresponding boundary region is lower than the boundary gradient threshold, it is determined that there is an oversegmentation, and the corresponding regions are merged. When there are multiple stable particle scale candidate points within a single particle region, and the scale difference value within the region is greater than the scale difference value threshold, it is determined that there is an undersegmentation, and the region is split a second time.

[0037] Step 5: Infrared particle segmentation image, processed by a pre-trained particle size regression model, yields the equivalent particle size of individual particles. Particle size statistics are then used to obtain solid waste particle size information. The particle size regression model can be selected according to requirements, including but not limited to implementation using a CNN model.

[0038] Then, based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, the particle adhesion assessment value is calculated. In this field, gravel particles with a particle size range of 2-5 mm are considered as fine particles. The formula for calculating the particle adhesion assessment value is as follows: , , , This is the evaluation value for particle adhesion. For the proportion of fine particles, For particle size distribution concentration, For particle size standard deviation, The median particle size is the normalized value for equivalent particle size. The smaller the median particle size, the more particles per unit area in a fixed field of view, making it easier for contact and adhesion to occur, and the more stable the solid waste. The median particle size is... For the first i The particle size of each particle, For the maximum particle size, These are infrared statistical modulation values. The local information entropy of the solid waste region in the infrared image. The maximum local information entropy is the maximum value. Lower local information entropy results in more uniform thermal textures and more noticeable adhesion. , , , These are the weights for the proportion of fine particles, the concentration of particle size distribution, the stability of solids, and the infrared statistical modulation.

[0039] Finally, based on the particle adhesion assessment value, the adhesion attribute is divided into low adhesion level (particle adhesion assessment value < first particle adhesion assessment value threshold), medium adhesion level (first particle adhesion assessment value threshold ≤ particle adhesion assessment value ≤ second particle adhesion assessment value threshold), and high adhesion level (particle adhesion assessment value > second particle adhesion assessment value threshold). The higher the particle adhesion assessment value, the higher the adhesion level.

[0040] When the adhesion attribute is classified as low adhesion level, medium adhesion level, and high adhesion level, the corresponding multimodal fusion operations S2, S3, and S4 are executed.

[0041] S2: When the solid waste is of low adhesion level, a channel weighted fusion method guided by infrared image is adopted to fuse the solid waste area of ​​the infrared image and the RGB image to obtain a multimodal fusion feature map, and then S5 is executed.

[0042] Low-adhesion particles typically have relatively obvious particle boundaries and limited contact areas. Illumination variations and background noise can severely affect segmentation accuracy. Infrared-guided channel weighted fusion can effectively suppress the influence of low-frequency illumination variations and maintain the clarity of particle edge information, avoiding missegmentation caused by background noise. At the same time, it can enhance texture and boundary clarity, improve particle recognition capabilities, and accurately distinguish particle structures, especially under uneven ambient lighting conditions.

[0043] The specific steps are detailed below.

[0044] Step 1: Divide the solid waste region of the infrared image into local windows, calculate the local gradient energy of the infrared temperature value in each window, normalize the local gradient energy to obtain the infrared thermal response instability map, which is used to characterize the stability of the infrared thermal response at different spatial locations.

[0045] Step 2: The infrared thermal response instability map is inversely smoothed to obtain an infrared component stability guide map. In the infrared component stability guide map, the larger the value, the more stable the infrared component is in that region and the better it reflects the material properties of solid waste. Global statistics are performed on the infrared component stability guide map to obtain an infrared stability description vector. Based on the infrared stability description vector, RGB channel weights and infrared channel weights are generated respectively. During this process, when the infrared stability is low, the RGB channel weights are increased and the infrared channel weights are suppressed based on preset channel weight rules, so that the infrared information does not only participate in fusion but also determines which is more reliable.

[0046] Step 3: Extract channel features from the solid waste region in the RGB and infrared images respectively. Channel feature extraction can be achieved through the channel attention mechanism to obtain RGB channel feature maps and infrared channel feature maps. Based on the RGB channel weights and infrared channel weights, perform a weighted summation of the RGB channel feature maps and RGB channel weights to obtain a multimodal fusion feature map.

[0047] The specific method for obtaining the solid waste region of an RGB image is as follows.

[0048] The RGB image is converted into a luminance component image, and then subjected to edge detection and edge-based smoothing to obtain an RGB smoothed image. The difference between the luminance component image and the RGB smoothed result is used as the RGB luminance residual image.

[0049] On the RGB luminance residual map, local frequency features are extracted using a sliding window method to calculate the RGB confidence map.

[0050] Specifically, based on the RGB luminance residual map, a fixed-size sliding window is deployed. A Fast Fourier Transform is performed on the area covered by each window to generate the corresponding local frequency power spectrum. From this spectrum, a preset mid-to-high frequency ring band region is selected (the radius of this ring band is preferably set to the range of 0.15π~0.45π of the normalized frequency). The energy proportion within the mid-to-high frequency ring band region is defined as the local mid-to-high frequency ring band energy ratio, thereby quantifying the significance of particle edges and textures. An integral operation is performed on the angular energy distribution of the mid-to-high frequency ring band region along the angular direction, and then the angular isotropic index is calculated using variance and mean, thus characterizing the uniformity of the local texture direction. Based on the high-frequency ring band energy ratio and the angular isotropic index (which can be achieved through weighted summation), an RGB confidence map is constructed. The RGB confidence map can intuitively reflect the probability that each pixel belongs to a solid waste region, accurately distinguishing solid waste regions with random particle texture characteristics from background regions with directional textures or smooth characteristics, thereby significantly reducing the probability of missegmentation in complex background environments.

[0051] The RGB image undergoes a logarithmic chromaticity transformation to reduce interference from light intensity fluctuations, resulting in a logarithmic chromaticity image. Local texture features are then extracted from the logarithmic chromaticity image, and texture complexity is calculated to obtain a texture stability map. Finally, the RGB confidence score and the texture stability map are weighted and fused to obtain an RGB comprehensive confidence map. The extraction of local texture features can be achieved using local binary mode.

[0052] Based on the RGB comprehensive confidence map, the foreground and background seeds corresponding to the image mask are established. Under the constraints of the foreground seed mask and the background seed mask, the RGB image is divided into regions to obtain the initial RGB segmentation result image. After morphological optimization processing, including filling holes, removing isolated regions and smoothing boundaries, the solid waste region of the RGB image is finally obtained.

[0053] S3: When the solid waste is of medium adhesion level, a frequency domain fusion method guided by RGB image is adopted to fuse the solid waste area of ​​the infrared image and the RGB image to obtain a multimodal fusion feature map. Then, S5 is executed to enhance the particle contact boundary and suppress the influence of uneven lighting or thermal field on segmentation and recognition. The specific steps are detailed as follows.

[0054] Step 1: Construct a particle boundary consistency guide map based on the solid waste region of the RGB image to characterize the stability of particle edges at different spatial scales in the RGB image.

[0055] Multi-scale edge detection is performed on the solid waste region of the RGB image to obtain RGB edge response maps at different spatial scales. At each scale, the edge direction consistency of the RGB edge response map is calculated to obtain the boundary consistency metric at each scale. The boundary consistency metrics at different scales are fused to generate an RGB particle boundary consistency guide map. The larger the value, the more stable and reliable the particle boundary at that location is at multiple scales.

[0056] Step 2: After converting the solid waste regions of the infrared and RGB images to the frequency domain, they are divided into low-frequency, mid-frequency, and high-frequency components according to the frequency radius. Based on the particle boundary consistency guide map, the RGB and infrared spectrum components of different frequencies are weighted and modulated to obtain the low-frequency, mid-frequency, and high-frequency modulation weights of the RGB and infrared spectrum components, respectively.

[0057] During weighted modulation, in the high-frequency band, based on the RGB particle boundary consistency guidance map and a preset high-frequency weight enhancement rule, the weight of the RGB high-frequency components is increased to enhance the distinction between particle boundaries and contact positions. In the low-frequency band, the weight of the RGB low-frequency components is decreased according to a preset low-frequency weight reduction rule, while the weight of the infrared low-frequency components is increased according to a preset low-frequency weight increase rule, to preserve the thermal component distribution and overall structural information of solid waste. In the mid-frequency band, based on the boundary consistency guidance map and a preset mid-frequency weight enhancement rule, the weights of the infrared mid-frequency components and RGB mid-frequency components are increased to avoid excessive boundary enhancement or loss of component information. By enhancing the mid- and high-frequency components, the boundary contrast of particles can be strengthened, making the identification of contacting or overlapping particles more accurate and avoiding blurred boundaries caused by contact.

[0058] Step 3: At the corresponding frequency position, the low-frequency component modulation weight, the mid-frequency component modulation weight, and the high-frequency component modulation weight are used to weight and superimpose the infrared spectrum components and RGB spectrum components of different frequencies to obtain the frequency domain fusion feature. After inverse frequency transformation, the multimodal fusion feature map in the spatial domain is obtained, which contains infrared component information and RGB particle boundary structure information.

[0059] S4: When the solid waste is of a high adhesion level, the particle center of the 3D point cloud model of the solid waste is used as the spatial anchor point. The solid waste region of the RGB image and infrared image is fused at the instance level to suppress large-area adhesion interference and enhance the separability of single particle instance features. This can ensure the accurate fusion of particle geometric information and image features, effectively avoid the influence of particle overlap or blurring, and obtain a multimodal fusion feature map. Then, proceed to S5.

[0060] Step 1: Downsample and denoise the 3D point cloud model of solid waste. Based on the extreme values ​​of point cloud density and / or curvature and / or local voxel occupancy, obtain the particle center point as the most stable common reference-spatial anchor point across modes. This is used for subsequent multimodal instance-level alignment, so that multimodal fusion does not depend on the visibility of 2D boundaries.

[0061] Step 2: Based on the calibration relationship between the 3D point cloud model and the RGB and infrared images, project the spatial anchor points onto the RGB and infrared images; with the projection position as the center, crop the corresponding solid waste area in the RGB and infrared images according to the preset spatial scale.

[0062] Step 3: For the solid waste regions in the RGB and infrared images, perform scale normalization and coordinate remapping respectively, and align them in the instance coordinate system; based on the spatial anchor point, calculate the radial distribution of the solid waste regions in the RGB and infrared images relative to the particle center, and assign corresponding geometric consistency weights; according to the geometric consistency weights of the RGB and infrared images, adjust the pixels of their respective solid waste regions in a channel-invariant manner to obtain the RGB modulation feature map and the infrared modulation feature map.

[0063] Step 4: Perform weighted fusion of the RGB modulation feature map and the infrared modulation feature map to generate multimodal fusion features based on particle instances; summarize the multimodal fusion features of all particle instances to form a multimodal fusion feature map.

[0064] S5. Map the infrared image and multimodal fusion feature map onto the three-dimensional point cloud model of solid waste to construct a three-dimensional solid waste particle model; based on the three-dimensional solid waste particle model, identify the particle type and particle size of each particle in the solid waste to form the particle identification result of solid waste.

[0065] First, infrared images and multimodal fusion feature maps are mapped onto a three-dimensional point cloud model of solid waste to construct a three-dimensional solid waste particle model.

[0066] Specifically, a planar spatial correspondence is established based on the calibration relationship between the 3D point cloud model and the RGB and infrared images. Based on the planar spatial correspondence, the infrared image and the multimodal fusion feature map are assigned to the point cloud of the 3D point cloud model to obtain a 3D solid waste particle model with multimodal attributes including infrared features, multimodal fusion features, and spatial features.

[0067] Then, based on the three-dimensional solid waste particle model, the particle type and particle size of individual particles in the solid waste are identified, forming the particle identification results of solid waste and calculating the particle size parameters of individual particles.

[0068] Specifically, the point cloud in the 3D solid waste particle model is initially clustered and corrected (including splitting or merging) to obtain single particle 3D instances; the geometric features, infrared features, and fusion attribute features of each 3D particle instance are obtained, input into a pre-trained classification model (preferably PointNet network), and the particle type and confidence score are output, along with the particle size of the corresponding single particle, to form the particle identification result; all particle identification results form the particle identification result of solid waste.

[0069] To improve the accuracy of the particle recognition results, this embodiment also includes obtaining the image texture quality value of the RGB image; determining whether the image texture quality value is greater than the image texture quality value threshold; if it is not greater than the threshold, performing image enhancement processing on the RGB image to obtain an RGB enhanced image, replacing the RGB image in S2, S3, or S4, and using it to perform the operations in S2, S3, or S4.

[0070] The image texture quality value is obtained based on the Laplacian variance, contrast estimation, noise estimation, edge complexity, and local information entropy of the stacked region segmentation map, and the calculation formula is as follows: , Image texture quality value, Let Laplace variance be the variance. , These represent the standard deviation and mean of pixels in an RGB image, respectively. For contrast estimation, For noise estimation, In RGB images (i, j The corresponding pixel grayscale value at position ) The Gaussian filter value, , These represent the total number of pixel positions in the horizontal and vertical directions, respectively. The edge complexity of the feature window region on the RGB image. The total number of pixels in the feature window region. For edge complexity, The local information entropy of the feature window region on the RGB image. , , , , They are the first weight, the second weight, the third weight, the fourth weight, and the fifth weight, respectively.

[0071] The image enhancement described above is used to improve the problems of blurred particle boundaries and uneven illumination in solid waste, enhancing the clarity of particle boundaries and texture contrast. Specifically, the RGB image is converted to the frequency domain. Based on the energy distribution and angular consistency characteristics of the frequency components, adaptive weighted modulation is applied to different frequency components to suppress low-frequency illumination variations and enhance mid-to-high-frequency particle edge information. The modulated spectrum is then converted back to the spatial domain to obtain the RGB enhanced image, thereby improving the clarity of particle boundaries and texture contrast. The angular consistency index is the ratio of the variance to the mean of the angular energy distribution.

[0072] The image enhancement operation described above can also be achieved by performing wavelet transform on the RGB image and then reconstructing the image.

[0073] This embodiment also provides a multimodal fusion-based intelligent identification system for solid waste particles, used to implement the above-mentioned multimodal fusion-based intelligent identification method for solid waste particles, including: The adhesion level generation module acquires infrared images of solid waste, performs particle segmentation and particle size statistics to obtain solid waste particle size information; based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the solid waste particle size information, it calculates particle adhesion assessment values; according to the particle adhesion assessment values, the adhesion attributes are divided into low adhesion level, medium adhesion level, and high adhesion level, corresponding to the execution of the first fusion module, the second fusion module, and the third fusion module, respectively; the higher the particle adhesion assessment value, the higher the adhesion level; The first fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a channel-weighted fusion method guided by the infrared image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The second fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a frequency domain fusion method guided by the RGB image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The third fusion module is used to use the particle center of the three-dimensional point cloud model of solid waste as a spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and execute the particle recognition result generation module. The particle identification result generation module is used to map infrared images and multimodal fusion feature maps onto a three-dimensional point cloud model of solid waste to construct a three-dimensional solid waste particle model. Based on the three-dimensional solid waste particle model, the particle type and particle size of each individual particle in the solid waste are identified, thus forming the particle identification result of the solid waste.

[0074] This embodiment also provides a multimodal fusion-based intelligent identification device for solid waste particles, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-described multimodal fusion-based intelligent identification method for solid waste particles.

[0075] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described intelligent identification method for solid waste particles based on multimodal fusion.

[0076] This embodiment provides a multimodal fusion-based intelligent identification method for solid waste particles. First, by performing particle segmentation and particle size statistics on infrared images of solid waste, features such as the proportion of fine particles, particle size distribution concentration, and equivalent particle size are calculated to obtain particle adhesion assessment values, and solid waste is classified into low, medium, and high adhesion levels accordingly. Then, for different adhesion levels, infrared-guided channel-weighted fusion, RGB-guided frequency domain fusion, or instance-level alignment fusion based on 3D point cloud spatial anchor points are used respectively to fully leverage the advantages of infrared images, RGB images, and 3D geometric information in their respective applicable scenarios, obtaining multimodal fusion features with stronger discriminative power. Finally, the infrared images and multimodal fusion features are mapped onto a 3D point cloud model to construct a 3D solid waste particle model, simultaneously obtaining accurate geometric scale and material features at the particle instance level, thereby significantly improving the accuracy of solid waste particle type identification and particle size measurement and the overall identification efficiency while reducing invalid calculations and missegmentation.

[0077] While exemplary embodiments of the invention have been described herein, many other variations or modifications conforming to the principles of the invention can be directly determined or derived from the disclosure of the invention without departing from its spirit and scope. Therefore, the scope of the invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A method for intelligent identification of solid waste particles based on multimodal fusion, characterized in that, This includes the following operations: S1: Acquire infrared images of solid waste, and obtain solid waste particle size information through particle segmentation and particle size statistics; Based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the particle size information of solid waste, the particle adhesion assessment value is calculated. According to the particle adhesion assessment value, the adhesion attribute is divided into low adhesion level, medium adhesion level, and high adhesion level, which correspond to S2, S3, and S4 respectively. The larger the particle adhesion assessment value, the higher the adhesion level. S2: Using a channel-weighted fusion method guided by infrared images, the solid waste regions of the infrared images and RGB images are fused to obtain a multimodal fusion feature map, and then S5 is executed; S3: Using a frequency domain fusion method guided by RGB images, the solid waste region of the infrared image and the RGB image are fused to obtain a multimodal fusion feature map, and then S5 is executed; S4: Using the particle center of the 3D point cloud model of solid waste as the spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and then execute S5. S5: Map infrared images and multimodal fusion feature maps onto a 3D point cloud model of solid waste to construct a 3D solid waste particle model; Based on a three-dimensional solid waste particle model, the particle type and size of each individual particle in the solid waste are identified, forming the particle identification results of the solid waste.

2. The method for intelligent identification of solid waste particles based on multimodal fusion according to claim 1, characterized in that, The specific operation of S2 is as follows: The solid waste region of the infrared image is divided into local windows. The local gradient energy of the infrared temperature value is calculated in each window. After normalization, the infrared thermal response instability map is obtained. The infrared thermal response instability map is decoupled and smoothed to obtain the infrared component stability guide map; global statistics are performed on the infrared component stability guide map to obtain the infrared stability description vector. Based on the infrared stability description vector, RGB channel weights and infrared channel weights are generated respectively. Channel features were extracted from the solid waste region in the RGB and infrared images respectively to obtain RGB channel feature maps and infrared channel feature maps; Based on the RGB channel weights and infrared channel weights, the RGB channel feature maps are weighted and summed with the RGB channel weights to obtain a multimodal fusion feature map.

3. The intelligent identification method for solid waste particles based on multimodal fusion according to claim 2, characterized in that, The method for obtaining solid waste regions in infrared images is as follows: The infrared image is scale-smoothed to obtain the background temperature field. The difference between the infrared image and the background temperature field is calculated to obtain the infrared foreground enhancement map. After normalization, the probability value of each pixel belonging to the solid waste region is calculated based on the temperature amplitude and local temperature gradient. The probability values ​​of all pixels are combined to form the infrared foreground probability map. Based on the histogram distribution of probability values ​​in the infrared foreground probability map, the foreground segmentation threshold is adaptively determined. Pixels with probability values ​​greater than the probability value threshold are marked as solid waste regions, and the remaining pixels are marked as background regions, thus obtaining the initial solid waste region map. Connectivity analysis is performed on the initial solid waste region map to remove connected regions with an area smaller than a preset area threshold. Morphological closing operations are then performed on the remaining regions to obtain the solid waste region in the infrared image.

4. The intelligent identification method for solid waste particles based on multimodal fusion according to claim 1, characterized in that, The specific operation of S3 is as follows: Based on the solid waste region of the RGB image, a particle boundary consistency guidance map is constructed. After converting the solid waste region of the infrared image and the RGB image to the frequency domain, they are divided into low-frequency, mid-frequency and high-frequency components according to the frequency radius. According to the particle boundary consistency guidance map, the RGB spectrum components and infrared spectrum components of different frequencies are weighted and modulated to obtain the low-frequency component modulation weight, mid-frequency component modulation weight and high-frequency component modulation weight of the RGB spectrum components and infrared spectrum components, respectively. At the corresponding frequency position, the infrared spectrum components of different frequencies are weighted and superimposed with the RGB spectrum components to obtain the frequency domain fusion feature. After inverse frequency transformation, a multimodal fusion feature map is obtained.

5. The intelligent identification method for solid waste particles based on multimodal fusion according to claim 1, characterized in that, In S5, the point cloud in the three-dimensional solid waste particle model is initially clustered and corrected to obtain single particle three-dimensional instances; the geometric features, infrared features, and fusion attribute features of each three-dimensional particle instance are obtained, input into the pre-trained classification model, and the particle type and confidence score are output. Together with the particle size parameter of the corresponding single particle, the particle recognition result is formed; all particle recognition results form the particle recognition result of solid waste.

6. The intelligent identification method for solid waste particles based on multimodal fusion according to claim 1, characterized in that, It also includes obtaining the image texture quality value of the RGB image; and determining whether the image texture quality value is greater than the image texture quality value threshold. If the value is not greater than the specified value, perform image enhancement processing on the RGB image to obtain an enhanced RGB image, which is then used to perform the operations in S2, S3, or S4.

7. The intelligent identification method for solid waste particles based on multimodal fusion according to claim 6, characterized in that, The image enhancement process involves converting the RGB image to the frequency domain, adaptively weighting the different frequency components based on their energy distribution and angular consistency characteristics, and then converting the modulated spectrum back to the spatial domain to obtain the enhanced RGB image.

8. A multimodal fusion-based intelligent identification system for solid waste particles, used to implement the multimodal fusion-based intelligent identification method for solid waste particles as described in claim 1, characterized in that, include: The adhesion level generation module acquires infrared images of solid waste, and obtains solid waste particle size information through particle segmentation and particle size statistics. Based on the proportion of fine particles, particle size distribution concentration, and equivalent particle size statistical characteristics in the particle size information of solid waste, a particle adhesion assessment value is calculated. According to the particle adhesion assessment value, the adhesion attribute is divided into low adhesion level, medium adhesion level, and high adhesion level, and the first fusion module, the second fusion module, and the third fusion module are executed respectively. The larger the particle adhesion assessment value, the higher the adhesion level. The first fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a channel-weighted fusion method guided by the infrared image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The second fusion module is used to fuse the solid waste region of the infrared image and the RGB image using a frequency domain fusion method guided by the RGB image to obtain a multimodal fusion feature map and execute the particle recognition result generation module. The third fusion module is used to use the particle center of the three-dimensional point cloud model of solid waste as a spatial anchor point, perform instance-level alignment and fusion of the solid waste region in the RGB image and infrared image to obtain a multimodal fusion feature map, and execute the particle recognition result generation module. The particle identification result generation module is used to map infrared images and multimodal fusion feature maps onto a three-dimensional point cloud model of solid waste to construct a three-dimensional solid waste particle model. Based on the three-dimensional solid waste particle model, the particle type and particle size of each individual particle in the solid waste are identified, thus forming the particle identification result of the solid waste.

9. A smart solid waste particle identification device based on multimodal fusion, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the intelligent identification method for solid waste particles based on multimodal fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the intelligent identification method for solid waste particles based on multimodal fusion as described in any one of claims 1-7.

Images