A garbage classification method and system fusing multi-modal features

Abstract

The application provides a garbage classification method and system fusing multi-modal features, relates to the field of computer vision, and solves the technical problems that the existing garbage classification method has poor adaptability to complex physical states and low recognition robustness due to single feature expression, and cannot realize fine classification and state evaluation. The method comprises the following steps: collecting multi-modal perception data of a garbage target to be classified; the multi-modal perception data at least comprises a high-resolution RGB image, a multispectral image, a depth image and a structured light image; inputting the multi-modal perception data into a pre-trained garbage classification model for processing, outputting a category, a spatial position and physical property information of the garbage target to be classified; and controlling an execution mechanism to complete a classification and sorting operation on the garbage target according to the output result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically a waste classification method and system that integrates multimodal features. Background Technology

[0002] With the acceleration of urbanization and the increasing demands for environmental protection, automated and intelligent waste sorting technologies have become a key focus in the waste management field. Currently, mainstream technical solutions mainly rely on machine vision, especially deep learning-based object detection and image classification models, to analyze RGB images of waste to achieve automatic identification and sorting.

[0003] However, deep learning-based machine vision solutions rely heavily on appearance features learned from RGB images. Therefore, when faced with real-world garbage flows, the severe deformation caused by squeezing and folding, or contamination and occlusion from oil or soup stains, significantly alters the visual characteristics of objects, leading to a substantial drop in model recognition accuracy. Furthermore, this approach suffers from dimensional limitations in feature perception, struggling to identify heterogeneous materials of the same color or the same material with different colors, and failing to perceive key attributes such as material and internal structure. For example, a yellow oil stain and a yellow plastic bottle may have highly similar color features in RGB images, but their materials are completely different, making them easily confused by the model. Conversely, a transparent mineral water bottle and a green beverage bottle, although both made of PET plastic, have significantly different appearances, potentially leading the model to misclassify them as different categories. Furthermore, in scenarios requiring refined sorting, such as further distinguishing recyclables by plastic resin type, or determining their degree of contamination and whether they are empty containers to decide on subsequent processing techniques, the existing system's classification granularity and intelligence level are insufficient to meet the precise needs of high-value resource recycling because single visual information cannot provide the material and physical state information required for such judgments. For example, existing waste sorting methods cannot reliably distinguish between PET (polyethylene terephthalate) and HDPE (high-density polyethylene) plastics, nor can they determine whether a milk carton has been crushed, whether there is residual liquid inside, or whether the oil stains on the surface of a plastic bottle have reached the level that require pre-cleaning. Summary of the Invention

[0004] This application provides a waste sorting method and system that integrates multimodal features, which solves the technical problems of existing waste sorting methods having poor adaptability to complex physical states, low recognition robustness, and inability to achieve refined classification and state assessment due to their single feature expression.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a waste classification method integrating multimodal features is provided, including: Collect multimodal sensing data of the waste to be classified; the multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images; The multimodal perception data is input into a pre-trained garbage classification model for processing; the garbage classification model is built based on a deep learning algorithm, including a dedicated feature extraction branch corresponding to each modality of data, and a cross-modal attention-guided fusion module for adaptive fusion of features from each branch; The cross-modal attention-guided fusion module calculates the confidence weight of each modal feature and performs weighted fusion of the extracted modal features based on the weight to obtain the fused features. Based on the fusion features, the decision head of the waste classification model outputs the category, spatial location, and physical attribute information of the waste to be classified. Based on the output results, the control actuator completes the sorting and classification operation of the waste target.

[0006] Based on the above technical solutions, the waste sorting method integrating multimodal features provided in this application achieves comprehensive acquisition of information on the appearance, material, three-dimensional geometry, and internal structure of waste targets by integrating perception data from four modalities: high-resolution RGB, multispectral, depth, and structured light. Secondly, the constructed deep learning model adopts a core architecture combining a dedicated feature extraction branch and a cross-modal attention-guided fusion module. The dedicated branch performs targeted feature extraction to preserve the unique information of different modal data. On the other hand, by introducing an attention mechanism to calculate confidence weights and performing adaptive weighted fusion, the contributions of each modality can be dynamically evaluated and integrated. When some modal information is disturbed, the more reliable modality is automatically prioritized, thereby improving the robustness and accuracy of the entire system in complex, unstructured real-world environments. Finally, the decision head of this method can simultaneously output the target's category, location, and physical attribute information, supporting not only basic waste sorting but also providing decision-making basis for subsequent refined sorting and state-dependent processing. This enables the entire sorting operation to move from simple differentiation to intelligent decision-making and refined management, improving the automation level and resource utilization efficiency of the waste treatment process.

[0007] Furthermore, the high-resolution RGB image is acquired using a high-resolution industrial camera; The multispectral image is a stack of multiple narrowband spectral channels, including the near-infrared band, used to identify the material type of the waste. The depth image is an image representing the distance information between each point on the target surface and the sensor, used to generate three-dimensional point cloud data of the garbage target, representing the geometric shape and spatial position of the garbage target; The structured light image is obtained by projecting a specific grating pattern onto the waste target and then collecting the grating deformation pattern. It is used to analyze the surface flatness and internal hollow state of the waste target.

[0008] Furthermore, the modal features include visual features, spectral features, depth geometric features, and structured light features, obtained through each of the dedicated feature extraction branches, including: The visual feature extraction branch extracts features from the high-resolution RGB image through the ConvNeXt network to obtain visual features; The spectral feature extraction branch extracts features from the multispectral image through the first convolutional neural network to obtain spectral features; The depth feature extraction branch extracts features from the 3D point cloud data converted from the depth image using the PointNet++ network to obtain depth geometric features. The structured light feature extraction branch extracts features from the structured light image through a second convolutional neural network to obtain structured light features.

[0009] Furthermore, the execution process of the cross-modal attention-guided fusion module includes: Self-attention calculations are performed on the visual features, spectral features, depth geometric features, and structured light features respectively to generate enhancement features for each modality; For each modality's enhancement features, the enhancement features of each modality are used as the query vector, and the concatenation result of the enhancement features of all other modalities is used as the key vector and value vector. Through the cross-attention mechanism, cross-attention calculation is performed on the query vector, key vector, and value vector to obtain the enhancement features of each modality that integrate information from other modalities. The average gradient magnitude and information entropy of each modality feature are calculated respectively, and the average gradient magnitude of each modality is input into the multilayer perceptron to generate the modality confidence weights corresponding to each modality. The enhanced features of each modality are multiplied by the corresponding modality confidence weights, and the weights are adjusted by a gating unit based on the Sigmoid function. Finally, all weighted features are summed to output the fused features.

[0010] Furthermore, the waste sorting model also includes a shared feature pyramid, the execution process of which includes: The fused features are received and, through multiple convolutional layers and downsampling operations, feature map sets {P2, P3, P4, P5} with different spatial resolutions are generated. Starting from feature map P5, upsampling is performed first. Then, the upsampling result is added element-wise to the features from the previous feature map P4 after channel adjustment to obtain the first layer of fused features. The channel adjustment is implemented through a 1×1 convolutional layer. The first layer is fused and upsampled, and then added element-wise to the features of the previous layer feature map P3 after channel adjustment to obtain the second layer fused features; The second layer of fused features is upsampled and added element-wise to the features of the previous layer feature map P2 after channel adjustment to obtain the third layer of fused features; The feature map P5, the first layer fusion feature, the second layer fusion feature, and the third layer fusion feature are processed again through 3×3 convolutional layers to output the enhanced multi-scale feature map {N2, N3, N4, N5}.

[0011] Furthermore, the decision head outputs the following information in parallel: The confidence score and bounding box coordinates of the main waste category; the main categories include recyclables, kitchen waste, hazardous waste, and other waste; Waste material subcategories include plastic, glass, metal, and paper; wherein, glass further includes transparent glass and colored glass, metal further includes aluminum and iron, and paper further includes corrugated cardboard, newspaper, book paper, and ordinary mixed paper. Waste status assessment information; the status assessment information includes a pollution level score and a label indicating whether the target waste is hollow.

[0012] Furthermore, the execution process of the decision head includes: Based on the multi-scale feature map, three parallel prediction branches are connected: an object detection branch, a material attribute branch, and a state evaluation branch; among which, The target detection branch pre-sets multiple anchor boxes of different scales and aspect ratios in the multi-scale diagnostic map, and obtains the main class confidence and bounding box coordinates of each anchor box through a convolutional layer; Based on the candidate regions determined by the target detection branch, the material property branch extracts the regional features of the candidate regions from the multi-scale feature map through the region of interest alignment operation. Then, the regional features are input into the fully connected layer for classification and recognition, and the probability distribution of the target waste as each material subcategory is output. The state assessment branch, based on the candidate region, extracts the features of each modal region aligned with the candidate region from the feature map of each modal feature through the region of interest alignment operation, and performs pollution degree scoring and hollow state judgment based on the features of each modal region, and outputs pollution degree score and label indicating whether the target waste is hollow.

[0013] Furthermore, the process of obtaining the pollution level score includes: Based on the spectral region features extracted from the candidate region, they are compared with the pre-stored reference spectral feature vectors of clean materials. The spectral distortion of the candidate region is obtained by calculating the cosine distance or mean square error between the two spectral feature vectors. The reference spectral feature vectors are obtained by pre-collecting the multispectral vectors of waste samples of each material sub-category and calculating the average value of the reflectance of the spectral frequency bands. Based on visual region features extracted from the same candidate region, color uniformity is evaluated by calculating the variance of the candidate region in the a and b channels of the CIELab color space, and the degree of texture anomaly is quantified by using the local binary mode variance or the contrast of the gray-level co-occurrence matrix. The average of the calculated results of color uniformity and texture anomaly is used as the visual pollution index. The spectral distortion and the visual pollution index are weighted, summed, and normalized to obtain a pollution score.

[0014] Furthermore, the label indicating whether the target waste is hollow is obtained by analyzing the consistency between the structured light region features and the depth geometric region features extracted from the candidate region, including: From the structured light region features, the gradient direction within a preset range is calculated to obtain the pattern features of the deformation of the target waste surface; the gradient direction is used to characterize the pattern features of bending, breaking or phase abrupt change of the grating stripes; The three-dimensional shape features of the target waste are obtained by calculating the rate of change of the normal to the surface within a preset range from the depth geometric region features; the rate of change of the normal is used to characterize the three-dimensional shape features with depressions, steep edges or closed cavities. Project the pattern features and the three-dimensional shape features onto the same two-dimensional coordinate space, and calculate the spatial overlap and geometric relationship between the pattern features and the three-dimensional shape features; If the spatial overlap between the pattern features and the three-dimensional shape features is higher than a preset threshold, and the geometric relationship meets the preset constraints, then the target waste is determined to be in a hollow state.

[0015] Furthermore, the geometric relationships include, but are not limited to, positional correlation, directional correlation, and correlation of change magnitude; wherein, The positional correlation is characterized by the average Euclidean distance between the feature points extracted from the pattern features and the feature points at the corresponding positions extracted from the three-dimensional shape features. The directional correlation is characterized by the angle between the principal direction of the deformation gradient in the pattern features and the principal direction of the depth gradient in the corresponding region of the three-dimensional shape features. The correlation of the change amplitude is characterized by the correlation coefficient between the phase change curve extracted from the pattern features and the depth change curve at the corresponding position in the three-dimensional shape features.

[0016] Furthermore, the step of controlling the actuator to complete the sorting and classification operation of the waste target based on the output results includes: Based on the main category confidence score and bounding box coordinates output by the decision head, the spatial location and main category of the target waste are determined. Based on the main category, the target waste is sorted into the corresponding collection bin or conveyor channel; For target waste whose main category is "recyclable", the waste is sorted to a more specific collection channel based on its material sub-category, and the sorting path is dynamically adjusted based on the status assessment information. If the pollution level of the target waste exceeds the preset pollution threshold, it will be sorted to the processing channel for centralized cleaning or specific disposal. If the target waste is determined to be in a hollow state, it is sorted to the hollow container channel, which is connected to compression equipment or residue monitoring equipment for internal residue detection and / or compression processing of the hollow container. If the target waste is determined to be non-hollow and the contamination score is below the threshold, the sorting channel will not be adjusted.

[0017] Secondly, this application provides a waste sorting system integrating multimodal features, including: a data acquisition module, an intelligent processing module, and a sorting execution module; wherein, The data acquisition module is used to acquire multimodal sensing data of the waste target to be classified. The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images. The intelligent processing module is used to input the multimodal perception data into a pre-trained waste classification model for processing. It uses the dedicated feature extraction branch in the waste classification model to obtain the features of each modality, and uses the cross-modal attention guidance fusion module of the waste classification model to calculate the confidence weight of each modality feature and perform weighted fusion. Based on the fused features, the decision head of the waste classification model outputs the category, spatial location and physical attribute information of the waste to be classified. Finally, it generates sorting control instructions based on the output information. The sorting execution module is used to receive the sorting control instructions and perform sorting operations on the waste targets.

[0018] Compared with the prior art, the beneficial effects of this application are: First, at the perception and feature level, by integrating four modal data of high-resolution RGB, multispectral, depth and structured light, and combining them with dedicated neural networks such as ConvNeXt and PointNet++ for feature extraction, this method achieves comprehensive and high-precision digital representation of the appearance texture, internal material, three-dimensional geometry and surface / internal physical state of waste targets. This not only solves the bottleneck of traditional single vision methods in dealing with surface contamination, severe deformation or heterogeneous objects of the same color, but also lays a solid data foundation for subsequent accurate classification and state assessment. Secondly, at the information fusion and decision-making level, the designed waste sorting model, through intramodal self-attention, cross-modal cross-attention, and dynamic reliability weight calculation based on feature quality, can intelligently evaluate and integrate heterogeneous information from different sensors. When some modalities are disturbed, it automatically relies on more reliable modalities, thereby improving the system's robustness and adaptability in complex, unstructured real-world environments. Simultaneously, the decision head adopts a multi-task parallel prediction architecture, outputting the main waste category, refined material subcategories, and physical attributes such as pollution level and hollow state in a single output. This achieves the integration of detection, classification, and state assessment, significantly improving processing efficiency and providing a complete decision-making basis for intelligent sorting. Finally, at the execution and application level, this method deeply couples the identification results with specific sorting operation rules. It can not only perform basic sorting of the four major categories, but also perform fine sorting based on material subcategories. Furthermore, it can combine dynamic information such as pollution scores and hollow states to adjust the sorting path in real time, which can effectively improve the purity and value of recyclables, reduce subsequent processing costs, and ultimately promote the waste treatment process towards a more efficient, intelligent, and resource-efficient direction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A system architecture diagram of a waste sorting system integrating multimodal features is provided for embodiments of this application; Figure 2 A flowchart illustrating a waste sorting method incorporating multimodal features, provided as an embodiment of this application; Figure 3 A flowchart illustrating another waste sorting method incorporating multimodal features provided in this application embodiment; Figure 4This is a flowchart illustrating another waste sorting method that integrates multimodal features, provided as an embodiment of this application. Detailed Implementation

[0021] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0022] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0023] The waste sorting method integrating multimodal features provided in this application embodiment can be applied to, for example... Figure 1 The waste sorting system shown integrates multimodal features, such as Figure 1 As shown, the system includes: a data acquisition module, an intelligent processing module, and a sorting execution module, which are connected in sequence via communication; wherein, The data acquisition module is used to collect multimodal sensing data of the waste to be classified. The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images. The intelligent processing module is used to input multimodal perception data into a pre-trained waste classification model for processing. It uses the dedicated feature extraction branch in the waste classification model to obtain the features of each modality, and guides the fusion module to calculate the confidence weight of each modality feature through the cross-modal attention of the waste classification model and performs weighted fusion. Based on the fused features, the decision head of the waste classification model outputs the category, spatial location and physical attribute information of the waste to be classified, and generates sorting control instructions based on the output information. The sorting execution module is used to receive sorting control instructions and perform sorting operations on waste targets.

[0024] To address the technical problems of existing waste sorting methods relying on single visual features, which result in low accuracy and weak robustness in complex scene recognition and the inability to achieve material identification and physical state assessment, this application provides a waste sorting method that integrates multimodal features. This method includes: Collect multimodal sensing data of the waste to be classified. The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images.

[0025] Multimodal perception data is input into a pre-trained waste classification model for processing. The model uses a dedicated feature extraction branch to obtain features of each modality. The model's cross-modal attention guides the fusion module to calculate the confidence weights of each modality feature and performs weighted fusion. Then, based on the fused features, the decision head of the waste classification model outputs the category, spatial location, and physical attribute information of the waste to be classified.

[0026] Based on the output results, the control actuator completes the sorting and classification of the waste targets.

[0027] Based on this, this method can comprehensively acquire information on the appearance, material, three-dimensional structure and physical state of waste, improve the stability of identification in complex environments, and support refined classification and intelligent sorting.

[0028] like Figure 2 As shown in the embodiment of this application, a waste sorting method integrating multimodal features is provided, including: S1. Collect multimodal sensing data of the waste to be classified.

[0029] The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images. High-resolution RGB images are used to present the color and texture details of the waste surface, serving as the basic basis for appearance recognition; multispectral images are used to capture the reflection characteristics of different bands, used to distinguish the material; depth images are used to record the distance information between the target and the acquisition device, used to reconstruct the three-dimensional spatial morphology; and structured light images reflect the surface flatness and internal cavity state of the object through the deformation of the projected pattern.

[0030] It should be noted that multimodal data needs to undergo denoising, normalization, and registration processing to ensure the accuracy of subsequent feature extraction and fusion. For example, an industrial camera array can be used to simultaneously acquire data from four different modalities, ensuring the spatial consistency of multi-source data on the same waste target.

[0031] S2. Input the multimodal sensing data into a pre-trained waste classification model for processing, and output the category, spatial location and physical attribute information of the waste to be classified.

[0032] Among them, physical property information is used to describe the state and material characteristics of the waste itself, which may include material type, degree of pollution, whether it is hollow, degree of deformation, surface integrity, etc.

[0033] In some implementations, if only garbage identification and classification are required, Convolutional Neural Networks (CNNs) and their variants, such as ResNet, VGG, and EfficientNet, can be used. These models excel at extracting high-level semantic features from images and performing classification. If localization is also required while identifying garbage, object detection networks, such as the YOLO series, Faster R-CNN, and SSD, can be used. These models can simultaneously output the object's category label and its bounding box coordinates in the image. Specifically, a classification network can be used to output category probabilities through feature extraction and fully connected layers, and a detection network can be combined to add a bounding box regression branch on top of the classification, simultaneously outputting the object's location and category. For example, a composite model based on the YOLO detection framework can be used, with multiple parallel branches extended to its backbone network to process multimodal data such as RGB and depth, and finally, a fusion module can be used to aggregate information and perform multi-task prediction.

[0034] It should be noted that the training of common waste classification models typically relies on large-scale, meticulously labeled image datasets. Training datasets mainly consist of massive amounts of RGB images of waste collected through web scraping, public datasets (such as TrashNet and TACO), and real-world scenarios. These image samples need to cover the diverse performance of target waste categories (such as recyclables, kitchen waste, etc.) under different lighting conditions, angles, backgrounds, and various typical physical states (such as crushed cans, stained bags, and partially obscured cardboard boxes). Furthermore, manual annotation is required to assign the correct category label and bounding box to each image or each waste target within it.

[0035] In some implementations, the model training process follows a standard supervised deep learning workflow, which may include: First, the labeled dataset is divided into training, validation, and test sets. Before training, image augmentation operations are typically performed, such as random cropping, rotation, color jittering, and adding noise, to expand the dataset and improve the model's robustness to scale, viewpoint, and some disturbances. During training, the model takes batches of images as input, calculates predictions through forward propagation, and then uses a loss function to measure the error between the prediction and the true label. Through backpropagation and gradient descent optimizers, the model iteratively updates its network weights to minimize the loss function. The entire training process is monitored on the validation set. Overfitting is prevented by observing metrics such as accuracy on the validation set, and transfer learning strategies may be employed, i.e., initializing model weights pre-trained on large general datasets such as ImageNet to accelerate convergence and improve performance. Finally, the model's generalization ability is evaluated on the test set to obtain key metrics such as classification or detection accuracy and recall. Nevertheless, the core capabilities of such models are limited by their training data—namely, the visual features of two-dimensional RGB images. This is essentially the fundamental reason why they struggle to cope with complex physical states and achieve refined classification.

[0036] S3. Based on the output results, control the actuator to complete the sorting and classification of the waste target.

[0037] In some implementations, different sorting strategies can be adopted based on the output results. For example, the system's main controller first generates basic drive instructions based on spatial location information and waste category to sort the target to the corresponding main channel. Then, it performs dynamic path planning by combining physical attribute information: for example, plastic bottles with "material type" of PET are sorted into PET-specific recycling bins; highly contaminated and recyclable targets are guided into the cleaning channel; and compressible containers are guided into the compression station. The actuator can be a high-speed robotic arm, a pneumatic nozzle array controlled by a programmable logic controller, or a push rod / baffle mechanism with switchable paths.

[0038] Based on the above technical solutions, this application provides a waste sorting method that integrates multimodal features. By systematically fusing visual, spectral, geometric, and structured light multidimensional information, a comprehensive and robust target feature representation system is constructed. Utilizing the adaptive fusion capability of deep learning models, the accuracy and stability of identifying contaminated, deformed, and complex material waste are improved. Simultaneously, this method incorporates physical property evaluation into the core output, enabling automated sorting systems to perform refined and differentiated operations based on material type and item state. This addresses the pain points of existing technologies, such as coarse classification granularity, poor environmental adaptability, and inability to support high-value recycling, providing an effective solution for achieving intelligent waste resource recovery.

[0039] In one possible implementation of the embodiments of this application, combined with Figure 2 The above S1 can be implemented through the following S101, S102, S103, and S104, which are explained in detail below: S101. High-resolution RGB images of the waste to be sorted are acquired using a high-resolution industrial camera.

[0040] Among them, high-resolution RGB images are used to obtain surface color, texture, outline and other appearance information of garbage targets, which is the basic data source for realizing garbage appearance feature perception and preliminary target positioning.

[0041] In some implementations, high-resolution RGB images are acquired by a high-resolution industrial camera fixed above the waste conveying channel, with the camera lens axis perpendicular to the waste conveying plane. Combined with uniform lighting, this ensures that the waste target has no obvious shadows, overexposure, or underexposure, and the image is clear and complete.

[0042] S102. Collect image stacks of multiple narrowband spectral channels, including the near-infrared band, to obtain a multispectral image of the waste target to be classified.

[0043] Among them, multispectral images are used to capture the spectral reflectance characteristics of different materials in different bands. They are key data for distinguishing material types such as plastic, glass, metal, and paper, and can effectively solve the identification confusion caused by different materials of the same color or the same material with different colors.

[0044] In some implementations, multispectral images are acquired synchronously by a multispectral imaging device, with channels covering the visible to near-infrared bands. The narrow-band channel images are stacked sequentially according to their bands to form a multi-channel image stack, where each channel image reflects the amount of reflected energy from the waste in its corresponding band. Because different materials exhibit significantly different spectral responses in specific bands, multispectral images can directly reveal the intrinsic material properties of the waste.

[0045] S103. Acquire depth images to characterize the distance information between each point on the target surface and the sensor, and generate three-dimensional point cloud data of the waste target to be classified from the depth images.

[0046] Among them, depth images and 3D point cloud data are used to characterize the 3D geometry, spatial location, stacking relationship and degree of deformation of waste targets. They are not affected by changes in color, stains and lighting, and can stably identify severely squeezed, folded and deformed waste.

[0047] In some implementations, depth images can be acquired using a depth camera or laser rangefinder. The value of each pixel in the depth image represents the actual distance between that location and the acquisition device. Then, coordinate transformation is performed based on the camera's intrinsic and extrinsic parameters and the depth values ​​to convert the depth image into point cloud data containing X, Y, and Z three-dimensional coordinates for subsequent depth geometric feature extraction.

[0048] It should be noted that 3D point cloud data can reconstruct the three-dimensional structure of garbage, improving the accuracy of target positioning and shape judgment.

[0049] S104. Project a specific grating pattern onto the waste target, collect the grating deformation pattern, and obtain a structured light image of the waste target to be classified.

[0050] Structured light imaging is used to analyze the surface smoothness, wrinkles, depressions, and internal hollow cavity status of waste targets, and is an important basis for determining whether containerized waste is hollow and whether there are internal residues.

[0051] In some implementations, a structured light projection device can project regular coded patterns such as sinusoidal gratings and Gray codes onto the waste target. Then, an industrial camera can capture the grating patterns that have been deformed after being modulated on the waste surface. Information such as stripe deformation, breakage, and phase abrupt changes can be used to reflect the surface undulations and internal cavity characteristics.

[0052] It should be noted that structured light images are highly sensitive to cavities, enclosed spaces, and subtle surface deformations, and can be cross-verified with depth geometric information, significantly improving the reliability of hollow state judgment.

[0053] Based on the above technical solution, S1 acquires high-resolution RGB images, multispectral images, depth images, and structured light images through a four-channel synchronous and independent acquisition process, forming comprehensive multimodal perception data covering appearance, material properties, three-dimensional geometry, and surface and internal physical states. These four types of data complement and verify each other, comprehensively overcoming the recognition deficiencies of single RGB visual data in contaminated, deformed, occluded, and heterogeneous scenarios with similar colors. This provides complete, reliable, and high-quality data support for subsequent specialized feature extraction, cross-modal adaptive fusion, and multi-task refined decision-making, enabling the system to maintain high recognition robustness and classification accuracy even in real-world, complex waste flow environments.

[0054] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S2 can be implemented through the following S201, S202, S203, S204, and S205, which are explained in detail below: S201. Through the dedicated feature extraction branch corresponding to each modal data, feature extraction is performed on the three-dimensional point cloud data and structured light image obtained by converting high-resolution RGB image, multispectral image, and depth image respectively, to obtain visual features, spectral features, depth geometric features and structured light features.

[0055] The dedicated feature extraction branch is designed to learn features specifically for the inherent characteristics of different modal data, preserving the unique and effective information in each modal data to the greatest extent possible. This avoids the loss of effective information when different types of data are mixed in the early stages, and provides high-quality, highly discriminative single-modal basic features for subsequent cross-modal feature fusion.

[0056] In some implementations, dedicated feature extraction branches include: The visual feature extraction branch uses the ConvNeXt network to extract global semantic information and local texture information from high-resolution RGB images, and outputs visual features that can completely characterize the appearance outline, surface texture and color distribution of garbage. The spectral feature extraction branch uses the first convolutional neural network to perform joint feature extraction of the channel dimension and spatial dimension of the multispectral image stack, and outputs spectral features that can reflect the differences in spectral reflectance of different materials. The depth feature extraction branch uses the PointNet++ network to extract local neighborhood features and global geometric features from the 3D point cloud data obtained by depth image conversion, and outputs depth geometric features that can characterize the 3D shape, spatial location and degree of deformation of the garbage. The structured light feature extraction branch uses a second convolutional neural network to extract stripe deformation and surface undulation related features from the structured light image, and outputs structured light features that can reflect the flatness, depressions, wrinkles and cavity structure of the waste surface.

[0057] It should be noted that the four dedicated branches operate in parallel and independently without interfering with each other. This allows them to adapt to the optimal receptive field, number of channels, and network depth for different modal data, ensuring that each modal feature achieves the best expression effect while improving the overall model's inference efficiency.

[0058] S202. Through the cross-modal attention-guided fusion module, self-attention enhancement, cross-attention fusion, modal trust weight calculation and weighted summation are performed on visual features, spectral features, depth geometric features and structured light features to obtain fused features.

[0059] Among them, the cross-modal attention-guided fusion module is used to dynamically evaluate the reliability and contribution of each modality feature in the current recognition task, automatically suppress interference modalities such as contamination, occlusion, and deformation, strengthen stable and effective modalities, and make the final fused features have strong robustness and high representation ability, thus solving the problem of decreased recognition accuracy when a single modality fails.

[0060] In some implementations, the internal workflow of the cross-modal attention-guided fusion module is as follows: First, self-attention calculation is performed on the features of the four modalities to enhance the key features within each modality and suppress redundant noise, thereby generating enhanced features for each modality. Then, the enhanced features of each modality are used as the query vector, and the enhanced features of all other modalities are concatenated to form the key vector and value vector. The cross-attention mechanism is used to complete the information interaction and complementarity between modalities, and the enhanced features of each modality are obtained by fusing information from other modalities. Next, the average gradient magnitude and information entropy of each modality feature are calculated. The average gradient magnitude is then input into the multilayer perceptron to generate modality confidence weights in the range of 0 to 1. The weights directly reflect the current confidence level of the modality. The formula for calculating the average gradient magnitude is as follows: In the formula, T is the average gradient magnitude of the current modality feature map, H and W are the height and width of the feature map, respectively, and F is the feature map matrix. F / x and F / y represents the gradient values ​​of the feature map in the horizontal and vertical directions, respectively. This formula is used to quantify the sharpness of the feature map; a higher gradient magnitude indicates a clearer feature outline and more reliable modal information. The modal trust weight calculation formula is: W=MLP(T); where W is the modal trust weight, and MLP is a multilayer perceptron, which is used to map the gradient magnitude to a learnable trust coefficient to achieve adaptive weight allocation. Finally, each modality enhancement feature is multiplied by its corresponding modality confidence weight, and the weights are adaptively adjusted using a sigmoid function-based gating unit. The weighted features are then summed dimension-wise to output the final fused feature. The calculation formula is as follows: In the formula, This represents the final fused feature output; m represents the modality index, the set. These correspond to four modalities: visual, spectral, depth, and structured light. This represents the enhanced feature of the m-th mode obtained after cross-modal cross-attention computation; This represents the confidence weight of the m-th mode; It is a learnable gating parameter vector; It is the Sigmoid activation function. This indicates element-wise multiplication; It should be noted that this fusion method can automatically reduce the weight of some modal information when the quality is extremely poor, so that the model can rely entirely on other reliable modalities to complete the recognition, which significantly improves the adaptability in complex real-world scenarios.

[0061] For example, when the surface of the garbage is covered with a lot of soup, causing the RGB image features to fail, the average gradient magnitude of the visual features is significantly reduced, and the corresponding weights are automatically lowered. The model then relies on spectral features to distinguish materials and on depth and structured light features to determine shape and cavity state, ensuring stable and accurate recognition results.

[0062] S203. Multi-scale downsampling, cross-layer upsampling fusion, and convolution optimization are performed on the fused features through the shared feature pyramid to output the enhanced multi-scale feature map {N2,N3,N4,N5}.

[0063] Among them, the shared feature pyramid is used to decompose the fused features into feature maps with different spatial resolutions, taking into account the feature representation needs of both large and small waste, improving the model's detection and classification accuracy for waste targets of different sizes and stacking states, and realizing cross-layer information flow and enhancement of features.

[0064] In some implementations, the workflow for sharing the feature pyramid content is as follows: First, after receiving the fused features, the shared feature pyramid generates a set of feature maps {P2, P3, P4, P5} with resolutions from high to low through continuous convolutional layers and downsampling operations. Then, upsampling is performed starting from the P5 feature map with the lowest resolution. The upsampling result is added element-wise to the feature map of P4 after adjusting the number of channels through a 1×1 convolutional layer to obtain the first layer of fused features. Next, the first layer of fused features is upsampled and added element by element to the features of the P3 feature map after channel adjustment to obtain the second layer of fused features; The second layer of fused features is then upsampled again and added element by element to the features of the P2 feature map after channel adjustment to obtain the third layer of fused features; Finally, P5, the first layer fusion feature, the second layer fusion feature, and the third layer fusion feature are processed through 3×3 convolutional layers to remove aliasing effects and optimize feature representation, outputting the final enhanced multi-scale feature map {N2, N3, N4, N5}.

[0065] It should be noted that multi-scale feature maps can cover the full range of recognition scenarios, from small pieces of trash to large waste, effectively solving the problems of missed detections and false detections caused by large differences in trash size and stacking occlusion. For example, the N2 high-resolution feature map can be responsible for detecting small target trash such as bottle caps and paper scraps, the N5 low-resolution feature map is responsible for detecting large target trash such as cardboard boxes and large plastic buckets, and the intermediate-scale feature map is adapted to trash of regular size, achieving full-scene coverage.

[0066] S204. Through three parallel branches of the decision head—target detection branch, material attribute branch, and state evaluation branch—the main garbage category, bounding box coordinates, material sub-category, pollution level score, and hollow state label are output simultaneously based on multi-scale feature maps.

[0067] The decision head adopts a multi-task parallel architecture, which is used to achieve integrated output of target positioning, major category classification, fine material identification and physical state assessment, avoiding efficiency loss caused by multiple reasoning, providing the sorting execution agency with complete and fine decision-making basis, and improving the intelligence and precision of waste sorting.

[0068] In some implementations, the internal workflow of the decision head includes: The object detection branch pre-sets anchor boxes of various scales and aspect ratios on multi-scale feature maps, and outputs the confidence score of the main waste category and the bounding box coordinates of each anchor box through convolutional layers. The main categories include recyclables, kitchen waste, hazardous waste, and other waste. Based on the candidate regions determined by the target detection branch, the material attribute branch accurately extracts candidate region features from the multi-scale feature map through the region of interest alignment operation. The region features are then input into the fully connected layer for classification and recognition, and the probability distribution of material subcategories such as plastic type, glass type, metal type, and paper type is output. Glass type further distinguishes between transparent glass and colored glass, metal type further distinguishes between aluminum and iron, and paper type further distinguishes between corrugated cardboard, newspaper, book paper and ordinary mixed paper. The state assessment branch is also based on candidate regions. It extracts corresponding region features from the original feature maps of each modality through region of interest alignment operation, and completes the pollution score calculation and hollow state judgment respectively. It outputs the pollution score in the range of 0 to 1 and a binary label to indicate whether it is hollow.

[0069] It should be noted that the three branches share multi-scale features, which can reduce the amount of computation and inference time, and meet the real-time requirements of automated sorting lines.

[0070] For example, for a transparent mineral water bottle, the decision head can simultaneously output the main category as recyclable, the bounding box coordinates, the material subcategory as PET plastic, the pollution score as 0.15, and the hollow state as yes.

[0071] S205. Calculate the pollution level score based on spectral and visual characteristics, determine the hollow state based on the consistency of structured light characteristics and depth geometric characteristics, and complete the accurate assessment of the physical properties of the waste.

[0072] Among them, the contamination score is used to quantify the degree of surface contamination of waste, providing a basis for judging whether pre-treatment cleaning is required; the hollow state judgment is used to identify container waste such as bottles, cans, and boxes, providing guidance for subsequent emptying, compression and other processing processes. Together, they determine the refined sorting path of recyclables.

[0073] In some implementations, the pollution level score is obtained by weighting spectral distortion and visual pollution index. First, the spectral distortion is calculated by taking the cosine distance between the spectral characteristics of the candidate region and the reference spectral characteristics of the clean material. The reference spectral characteristics are obtained by averaging the multispectral reflectance of various types of clean waste samples collected beforehand. The calculation formula is as follows: ; In the formula For spectral distortion, The spectral vector of the region to be measured. The reference spectral vector for the clean sample; Subsequently, the variance of the CIELab color space a and b channels was calculated in the visual features to evaluate color uniformity, and the contrast ratio of the gray-level co-occurrence matrix was used to quantify the degree of texture anomalies. The average of the two values ​​was then used to obtain the visual pollution index. ; Next, the spectral distortion and visual pollution index are weighted, summed, and normalized to obtain a pollution score in the range of 0 to 1. α and β are weighting coefficients.

[0074] The process of determining the hollow state includes: First, it is necessary to ensure that the structured light features and depth geometric features extracted from the same candidate region of interest (ROI) are strictly aligned spatially. Assume that the structured light feature maps have been extracted using the ROIAlign operation. and deep point cloud features Extract the feature set of the corresponding region. For consistency analysis, the 3D point cloud features need to be... Projected onto the same imaging plane as the 2D structured light image. Let the camera intrinsic parameter matrix be K, and for a given 3D point... Projection point on the image plane From the formula The calculation shows that this projection establishes a correspondence between each 3D point and a 2D pixel, forming a registered feature pair. ,in This is the projected depth feature map.

[0075] Next, structured light pattern features are extracted. In structured light images, the distortion of grating fringes (bending, breakage, phase abrupt changes) directly reflects surface geometric changes. Within the registered Region of Interest (ROI), the computational structured light image is obtained using phase-shifting or Fourier transform profilometry. Phase diagram Pattern characteristics The phase gradient field can be calculated To represent: ; gradient direction and amplitude Together, they describe the local patterns of stripe deformation. For the edges of hollow containers or the recessed areas of labels, the gradient direction changes drastically, and the amplitude also increases significantly.

[0076] Then, the depth-based 3D shape features are extracted. Geometric features characterizing the surface's unevenness are calculated from the projected and aligned 3D point cloud data. Here, the surface normal change rate (i.e., curvature correlation metric) is used as the 3D shape feature. First, calculate the normal vector of each point in the point cloud. For point Calculate the covariance matrix within its neighborhood and perform eigenvalue decomposition to obtain the eigenvalues. Surface curvature An approximate estimate is: Among these, high curvature regions (such as depressions and edges) correspond to locations where hollow structures may cause abrupt surface changes. Simultaneously, the standard deviation of the normal vector within the neighborhood can be calculated to quantify the rate of change of the normal.

[0077] Next, the extracted 2D pattern features With 3D shape features Similarity measurement is performed within the same registered two-dimensional coordinate space: First, calculate the spatial overlap. Define a binarized feature mask. and , among which when or When the mask value is 1, it is 0 otherwise. , This is an empirical threshold. Spatial overlap. Calculated using the intersection-over-union ratio (IoU): In the formula This indicates the number of pixels with a value of 1 in the mask. High overlap indicates that the stripe distortion regions observed by structured light and the geometric abrupt change regions perceived by depth sensing are spatially highly overlapping, which is a strong indication of a hollow structure.

[0078] Then, perform multi-dimensional geometric relationship consistency verification: 1. Locational Correlation: Within the overlapping region, extract a set of significant feature points (such as phase abrupt change points, curvature maxima). Let's assume... and The feature point sets extracted are respectively and Find corresponding point pairs using nearest neighbor matching (such as FLANN). Then, use the average Euclidean distance between the matched point pairs. Measuring location correlation : ; Where N is the number of matching point pairs, This is the scale parameter. The closer to 1, the higher the positional consistency.

[0079] 2. Directional Correlation: Compare the direction of the structured light phase gradient at corresponding feature points. The principal curvature directions of the surface calculated in the depth features (The projection of the eigenvector direction corresponding to the largest eigenvalue onto the image plane). Then, the directional correlation is represented by the average cosine of the angles between the directions at these corresponding points. : .

[0080] 3. Correlation of variation amplitude: Phase change curves are obtained along a sampling profile line that passes through the feature region. and depth change curve (s is the curve parameter). Calculate the normalized cross-correlation coefficient (NCC) between the two curves as the correlation of their magnitude of change. : ;in , This is the average value. The higher the value, the more synchronized the changes in surface depth and the phase changes in grating fringes are in terms of their trends.

[0081] The final determination of the hollow state is a multi-condition decision-making process. This is achieved by setting a spatial overlap threshold. Position, direction, and amplitude correlation thresholds , , To make a comparison and judgment: A candidate target garbage is considered to be in a "hollow state" if and only if all of the following conditions are met: .

[0082] This decision-making logic ensures that the system only confirms the existence of an internal cavity structure in a target when the surface optical deformation pattern revealed by structured light features and the three-dimensional geometric shape features revealed by depth features exhibit a high degree of consistency in space, position, orientation, and magnitude of change. This significantly reduces misjudgments caused by surface mapping, single-viewpoint occlusion, or noise. This decision result, along with the contamination score, will be used to guide subsequent refined sorting processes.

[0083] Based on the above technical solutions, S2 completes the entire reasoning process from multimodal perception data to classification, localization, and state assessment results through five steps: dedicated feature extraction, cross-modal attention adaptive fusion, multi-scale feature pyramid enhancement, parallel output of multi-task decision heads, and refined physical attribute evaluation. The dedicated branch design fully preserves the unique information of each modality; cross-modal attention fusion dynamically improves robustness in complex scenarios; the multi-scale pyramid adapts to full-size waste recognition; the multi-task decision head achieves integrated and efficient output; and physical attribute evaluation provides the core basis for refined sorting. The overall model architecture balances recognition accuracy, robustness, and real-time performance, fundamentally solving the recognition deficiencies of traditional single-vision solutions in scenarios involving pollution, deformation, heterogeneous materials of the same color, and heterogeneous materials of the same color. This enables the waste sorting system to adapt to real and complex waste flows and meet the needs of refined recycling and intelligent processing of high-value resources.

[0084] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S3 can be specifically implemented through the following S301, S302, S303, and S304, which are explained in detail below: S301. Based on the main category confidence score and bounding box coordinates output by the decision head, determine the spatial location and main category of the target waste.

[0085] Among them, spatial location and main category are the basic basis for sorting execution, which are used to guide the execution agency to complete accurate positioning and major category sorting, avoid missorting and omissions, and ensure the stability and reliability of the basic classification process.

[0086] In some implementations, the system reads the bounding box coordinates output by the target detection branch, converts the image coordinates into spatial coordinates recognizable by the sorting actuator, and simultaneously reads the main category confidence score. When the confidence score is higher than a preset recognition threshold, the category is deemed valid, and the target waste is classified into the corresponding category among recyclables, kitchen waste, hazardous waste, and other waste. If the confidence score is lower than the preset threshold, the target waste is marked as an object awaiting review and enters the manual review channel to avoid low-confidence results affecting the overall sorting accuracy.

[0087] S302. Based on the main category of the target waste, control the actuator to sort the waste to the corresponding collection bin or conveyor channel.

[0088] Among them, the main category sorting is the basic process of waste classification, which is used to achieve the initial separation of the four major categories of waste, laying the foundation for subsequent resource recovery and harmless disposal.

[0089] In some implementation methods, the actuator performs corresponding actions according to the main category determined by S301. Recyclables enter the main recyclables channel, kitchen waste enters the dedicated kitchen waste channel, hazardous waste enters the sealed hazardous waste collection bin, and other waste enters the other waste collection channel. The different channels are independent of each other to avoid cross-contamination of various types of waste. The actuators include, but are not limited to, robotic arms, pneumatic push rods, swing valve guide devices, belt sorting machines, etc., which can be flexibly selected according to the on-site working conditions.

[0090] It should be noted that the main category sorting process has the highest priority. No matter how the subsequent refined sorting is adjusted, the correct classification of the main category must be ensured. This is a basic requirement that meets the national standards for waste classification.

[0091] S303. For target waste whose main category is recyclable, sort the waste to the corresponding sub-collection channel based on the material sub-category.

[0092] Among them, material subdivision sorting is used to achieve accurate sorting of high-value recyclables, improve the purity and recycling value of recycled resources, and solve the problem that traditional systems can only sort by broad categories and cannot sort finely.

[0093] In some implementations, the system reads the probability distribution of material subcategories output by the material attribute branch, selects the category with the highest probability as the final material type, and further subdivides and sorts them according to plastic type, glass type, metal type, and paper type. Plastic type is further subdivided into resin types such as PET and HDPE, glass type into transparent glass and colored glass, metal type into aluminum and iron, and paper into corrugated cardboard, newspaper, book paper, and ordinary mixed paper. Different materials enter independent subdivided collection channels to facilitate subsequent professional recycling processing.

[0094] S304. By combining pollution level scores with hollow status labels, the sorting path of recyclables is dynamically adjusted to achieve intelligent and refined sorting.

[0095] Dynamic path adjustment is the core of intelligent sorting, used to adapt the optimal subsequent processing technology according to the physical state of the waste, thereby improving resource utilization and reducing processing costs.

[0096] In some implementations, the system reads the contamination score and hollow status label output by the status assessment branch and performs dynamic adjustments according to preset rules: When the pollution scores of plastic, glass and metal types are higher than the preset pollution threshold, the target waste is switched from the regular recycling channel to the waiting channel for centralized cleaning, decontamination or specific pretreatment. When the target waste is determined to be in a hollow state and the pollution score is lower than the preset pollution threshold, it is introduced into a special channel for hollow containers. This channel is connected to the residue monitoring equipment and the compression equipment. The internal liquid or residue is detected first, and then it is flattened and compressed to reduce the volume. When the target waste is not hollow and the pollution score is lower than the preset threshold, the original subdivided sorting channel remains unchanged, and it directly enters the resource recycling process.

[0097] It should be noted that the contamination threshold can be set individually according to the characteristics of different materials. For example, the contamination threshold for paper is lower, while the contamination threshold for plastic can be appropriately relaxed, so that the sorting strategy is more in line with the actual recycling process requirements.

[0098] For example, a mineral water bottle has a contamination score of 0.7, which is higher than the plastic contamination threshold of 0.6, and is determined to be a hollow container. It first enters the cleaning channel, and after cleaning, it enters the hollow container compression channel. A clean cardboard has a contamination score of 0.1, is not hollow, and directly enters the cardboard recycling channel.

[0099] Based on the aforementioned technical solution, S3 completes the entire process from identification results to automated sorting through location discrimination, major category sorting, material sub-segmentation, and dynamic path adjustment. This process ensures compliance based on the main category, enhances resource value through material sub-segmentation, and achieves intelligent disposal through dynamic adjustment of physical state. It deeply couples multimodal identification output with industrial-grade sorting execution, enabling the system not only to identify waste but also to make optimal processing decisions based on the identification results. Compared to traditional fixed-path sorting methods, this method significantly improves sorting accuracy, recyclable purity, and subsequent processing efficiency, upgrading waste classification from passive categorization to proactive intelligent decision-making. It fully adapts to real-world, complex waste flow scenarios, promoting the development of waste treatment towards high efficiency, precision, and resource recovery.

[0100] How this application works: This application first uses a multimodal sensing unit to simultaneously acquire high-resolution RGB images, multispectral images, depth images and structured light images of the waste target, and completes full information acquisition from four dimensions: appearance color texture, material spectral characteristics, three-dimensional geometry, and surface and cavity physical state, making up for the shortcomings of single RGB visual data in being unable to identify materials and being susceptible to pollution and deformation interference. The four types of data are then input into the deep learning model, where corresponding dedicated feature extraction branches extract visual features, spectral features, depth geometric features, and structured light features, respectively, preserving the unique information of each modality to the greatest extent. Next, a cross-modal attention-guided fusion module first performs self-attention enhancement and cross-modal cross-attention interaction on the features of each modality. Then, dynamic confidence weights are calculated based on the average gradient magnitude of the features, effectively enhancing the features and suppressing interfering features, outputting highly robust fused features. The fused features are enhanced at multiple scales through a shared feature pyramid to generate feature maps that are adapted to targets of different sizes. Then, a decision head composed of three parallel branches—target detection, material properties, and state assessment—simultaneously outputs the main waste category, spatial location, fine material category, pollution level score, and hollow state label. Finally, based on the above results, the sorting execution module first completes basic sorting according to the four main categories, further subdivides recyclables according to material, and dynamically adjusts the sorting path according to the degree of pollution and the hollow state. Highly polluted waste is guided to the cleaning channel and hollow containers are guided to the residue monitoring and compression channel. This realizes the full-process operation from multimodal perception, feature adaptive fusion, multi-task integrated decision-making to intelligent and refined sorting, maintaining high recognition accuracy and sorting stability in complex waste flow environments.

[0101] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0102] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and modifications.

Claims

1. A waste sorting method integrating multimodal features, characterized in that, include: Collect multimodal sensing data of waste to be sorted; The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images; The multimodal perception data is input into a pre-trained garbage classification model for processing; the garbage classification model is built based on a deep learning algorithm, including a dedicated feature extraction branch corresponding to each modality of data, and a cross-modal attention-guided fusion module for adaptive fusion of features from each branch; The cross-modal attention-guided fusion module calculates the confidence weight of each modal feature and performs weighted fusion of the extracted modal features based on the weight to obtain the fused features. Based on the fusion features, the decision head of the waste classification model outputs the category, spatial location, and physical attribute information of the waste to be classified. Based on the output results, the control actuator completes the sorting and classification operation of the waste target.

2. The waste sorting method integrating multimodal features according to claim 1, characterized in that, The high-resolution RGB image was acquired using a high-resolution industrial camera. The multispectral image is a stack of multiple narrowband spectral channels, including the near-infrared band, used to identify the material type of the waste. The depth image is an image representing the distance information between each point on the target surface and the sensor, used to generate three-dimensional point cloud data of the garbage target, representing the geometric shape and spatial position of the garbage target; The structured light image is obtained by projecting a specific grating pattern onto the waste target and then collecting the grating deformation pattern. It is used to analyze the surface flatness and internal hollow state of the waste target.

3. The waste sorting method integrating multimodal features according to claim 1, characterized in that, The modal features include visual features, spectral features, depth geometric features, and structured light features, which are obtained through the respective dedicated feature extraction branches, including: The visual feature extraction branch extracts features from the high-resolution RGB image through the ConvNeXt network to obtain visual features; The spectral feature extraction branch extracts features from the multispectral image through the first convolutional neural network to obtain spectral features; The depth feature extraction branch extracts features from the 3D point cloud data converted from the depth image using the PointNet++ network to obtain depth geometric features. The structured light feature extraction branch extracts features from the structured light image through a second convolutional neural network to obtain structured light features.

4. The waste sorting method integrating multimodal features according to claim 3, characterized in that, The execution process of the cross-modal attention-guided fusion module includes: Self-attention calculations are performed on the visual features, spectral features, depth geometric features, and structured light features respectively to generate enhancement features for each modality; For each modality's enhancement features, the enhancement features of each modality are used as the query vector, and the concatenation result of the enhancement features of all other modalities is used as the key vector and value vector. Through the cross-attention mechanism, cross-attention calculation is performed on the query vector, key vector, and value vector to obtain the enhancement features of each modality that integrate information from other modalities. The average gradient magnitude and information entropy of each modality feature are calculated respectively, and the average gradient magnitude of each modality is input into the multilayer perceptron to generate the modality confidence weights corresponding to each modality. The enhanced features of each modality are multiplied by the corresponding modality confidence weights, and the weights are adjusted by a gating unit based on the Sigmoid function. Finally, all weighted features are summed to output the fused features.

5. A waste sorting method integrating multimodal features according to claim 4, characterized in that, The waste sorting model also includes a shared feature pyramid, the execution process of which includes: The fused features are received and, through multiple convolutional layers and downsampling operations, feature map sets {P2, P3, P4, P5} with different spatial resolutions are generated. Starting from feature map P5, upsampling is performed first. Then, the upsampling result is added element-wise to the features from the previous feature map P4 after channel adjustment to obtain the first layer of fused features. The channel adjustment is implemented through a 1×1 convolutional layer. The first layer is fused and upsampled, and then added element-wise to the features of the previous layer feature map P3 after channel adjustment to obtain the second layer fused features; The second layer of fused features is upsampled and added element-wise to the features of the previous layer feature map P2 after channel adjustment to obtain the third layer of fused features; The feature map P5, the first layer fusion feature, the second layer fusion feature, and the third layer fusion feature are processed again through 3×3 convolutional layers to output the enhanced multi-scale feature map {N2, N3, N4, N5}.

6. The waste sorting method integrating multimodal features according to claim 1, characterized in that, The decision head outputs the following information in parallel: The confidence score and bounding box coordinates of the main waste category; the main categories include recyclables, kitchen waste, hazardous waste, and other waste; Waste material subcategories include plastic, glass, metal, and paper; wherein, glass further includes transparent glass and colored glass, metal further includes aluminum and iron, and paper further includes corrugated cardboard, newspaper, book paper, and ordinary mixed paper. Waste status assessment information; the status assessment information includes a pollution level score and a label indicating whether the target waste is hollow.

7. A waste sorting method integrating multimodal features according to claim 5, characterized in that, The execution process of the decision head includes: Based on the multi-scale feature map, three parallel prediction branches are connected: an object detection branch, a material attribute branch, and a state evaluation branch; among which, The target detection branch pre-sets multiple anchor boxes of different scales and aspect ratios in the multi-scale diagnostic map, and obtains the main class confidence and bounding box coordinates of each anchor box through a convolutional layer; Based on the candidate regions determined by the target detection branch, the material property branch extracts the regional features of the candidate regions from the multi-scale feature map through the region of interest alignment operation. Then, the regional features are input into the fully connected layer for classification and recognition, and the probability distribution of the target waste as each material subcategory is output. The state assessment branch, based on the candidate region, extracts the features of each modal region aligned with the candidate region from the feature map of each modal feature through the region of interest alignment operation, and performs pollution degree scoring and hollow state judgment based on the features of each modal region, and outputs pollution degree score and label indicating whether the target waste is hollow.

8. A waste sorting method integrating multimodal features according to claim 7, characterized in that, The process of obtaining the pollution level score includes: Based on the spectral region features extracted from the candidate region, they are compared with the pre-stored reference spectral feature vectors of clean materials. The spectral distortion of the candidate region is obtained by calculating the cosine distance or mean square error between the two spectral feature vectors. The reference spectral feature vectors are obtained by pre-collecting the multispectral vectors of waste samples of each material sub-category and calculating the average value of the reflectance of the spectral frequency bands. Based on visual region features extracted from the same candidate region, color uniformity is evaluated by calculating the variance of the candidate region in the a and b channels of the CIELab color space, and the degree of texture anomaly is quantified by using the local binary mode variance or the contrast of the gray-level co-occurrence matrix. The average of the calculated results of color uniformity and texture anomaly is used as the visual pollution index. The spectral distortion and the visual pollution index are weighted, summed, and normalized to obtain the pollution score.

9. A waste sorting method integrating multimodal features according to claim 7, characterized in that, The tag indicating whether the target waste is hollow is obtained by analyzing the consistency between the structured light region features and the depth geometric region features extracted from the candidate region, including: From the structured light region features, the gradient direction within a preset range is calculated to obtain the pattern features of the deformation of the target waste surface; the gradient direction is used to characterize the pattern features of bending, breaking or phase abrupt change of the grating stripes; The three-dimensional shape features of the target waste are obtained by calculating the rate of change of the normal to the surface within a preset range from the depth geometric region features; the rate of change of the normal is used to characterize the three-dimensional shape features with depressions, steep edges or closed cavities. Project the pattern features and the three-dimensional shape features onto the same two-dimensional coordinate space, and calculate the spatial overlap and geometric relationship between the pattern features and the three-dimensional shape features; If the spatial overlap between the pattern features and the three-dimensional shape features is higher than a preset threshold, and the geometric relationship meets the preset constraints, then the target waste is determined to be in a hollow state.

10. A waste sorting system integrating multimodal features, characterized in that, The system comprises a data acquisition module, an intelligent processing module, and a sorting execution module; among which, The data acquisition module is used to acquire multimodal sensing data of the waste target to be classified. The multimodal sensing data includes at least high-resolution RGB images, multispectral images, depth images, and structured light images. The intelligent processing module is used to input the multimodal perception data into a pre-trained waste classification model for processing. It uses the dedicated feature extraction branch in the waste classification model to obtain the features of each modality, and uses the cross-modal attention guidance fusion module of the waste classification model to calculate the confidence weight of each modality feature and perform weighted fusion. Based on the fused features, the decision head of the waste classification model outputs the category, spatial location and physical attribute information of the waste to be classified. Finally, it generates sorting control instructions based on the output information. The sorting execution module is used to receive the sorting control instructions and perform sorting operations on the waste targets.

Images