Waste material identification method and system based on multi-modal fusion

Abstract

This invention discloses a method and system for identifying waste materials based on multimodal fusion, belonging to the field of intelligent waste material identification technology. The method involves acquiring visible light image data, depth structure data, and near-infrared spectral response data of the waste materials to be identified; obtaining candidate regions from the visible light image data and extracting a set of morphological features based on the depth structure data; extracting material reflection curves from the near-infrared spectral response data; extracting texture distribution features and color stability features from the visible light image data and generating a set of structural consistency features based on the morphological features; constructing a fusion feature vector by performing cross-modal association mapping on multiple features and inputting it into a waste material identification model to obtain candidate identification results; identifying key feature subsets through local sensitivity analysis and reconstructing the weights of the fusion features; and performing classification again to output the final identification result. This invention can effectively improve the accuracy and stability of waste material identification in complex recycling scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent identification technology for waste materials, specifically to a waste material identification method and system based on multimodal fusion. Background Technology

[0002] In the recycling and sorting of recyclable resources, large quantities of waste materials enter the identification stage through mixed stacking or high-speed conveying. Some of these waste materials undergo deformation, corrosion, or adherence to residues during long-term use or storage. For example, flattened metal cans and coated plastic containers exhibit highly similar appearance and texture, while paper packaging covered in oil stains can be subtly confused with plastic film in visible light images. Existing identification methods typically rely on a single visual model or simple multi-source information stitching. These methods struggle to reliably extract key discriminative features when faced with partial occlusion, material aging, or fine-grained category differences, especially with waste materials exhibiting minimal local microstructural changes, leading to misjudgments or unstable identification. Therefore, developing a fusion identification method that reflects both the microstructural changes and material properties of waste materials in complex recycling environments is crucial for improving the reliability of automated sorting systems. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for identifying waste materials based on multimodal fusion, so as to overcome the shortcomings of the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a waste material identification method based on multimodal fusion, comprising: Acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S; The visible light image data I is subjected to target detection and instance segmentation to obtain a candidate region set R of waste materials. Based on the candidate region set R, the corresponding three-dimensional morphological parameters are extracted from the depth structure data D to form a morphological feature set F1. Based on the candidate region R, extract the material reflection curve from the near-infrared spectral response data S, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2; Based on the candidate region set R, texture distribution features and color stability features are extracted from the visible light image data I, and the structural consistency feature set F3 is generated by combining the morphological feature set F1. The morphological feature set F1, the material variation feature set F2, and the structural consistency feature set F3 are subjected to cross-modal correlation mapping to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials. The fused feature vector F is input into the waste material identification model for classification and discrimination to obtain candidate identification results; Based on the candidate identification results, a local sensitivity analysis is performed on the fused feature vector to identify the key feature subset that has the greatest impact on the classification results. The weights of the fused feature vector are then reconstructed based on the key feature subset to form the updated fused features. The updated fusion features are then input into the recognition model for discrimination, and the final waste material category recognition result is output.

[0005] Preferably, target detection and instance segmentation are performed on the visible light image data I, including: The visible light image data is normalized, and the waste material targets in the image are initially located using a target detection model based on a convolutional neural network to obtain several target candidate boxes. Pixel-level instance segmentation is performed on the target candidate boxes to generate corresponding target segmentation masks, and the candidate region set of waste materials is determined based on the target segmentation masks.

[0006] Preferably, the method for obtaining the morphological feature set F1 is as follows: the candidate region set is mapped to the corresponding depth structure data, the depth value of each pixel in the candidate region is extracted by pixel coordinate matching, and the three-dimensional point cloud data corresponding to the candidate region is constructed; the three-dimensional point cloud data is subjected to surface fitting and spatial distribution statistical analysis to calculate the three-dimensional morphological parameters including volume parameters, surface curvature parameters and shape dispersion parameters, thereby forming the morphological feature set of waste materials.

[0007] Preferably, calculating the local gradient change characteristics of the material reflection curve includes: extracting spectral reflectance values ​​within the corresponding wavelength range based on the pixel positions of the candidate region in the near-infrared spectral response data, and constructing the material reflection curve of the waste material according to wavelength order; performing spectral smoothing on the material reflection curve, and fitting adjacent wavelength sampling points using a sliding window polynomial fitting method to obtain a smoothed material reflection curve; constructing a fractional derivative spectral analysis model based on the smoothed material reflection curve, and performing fractional derivative operations on the material reflection curve using the Glenwald-Letnikov fractional derivative calculation method to obtain local gradient change values ​​at different wavelength positions; calculating gradient change intensity parameters and gradient change distribution parameters based on the local gradient change values ​​corresponding to each wavelength position, and combining the gradient change intensity parameters and gradient change distribution parameters to form a set of material change characteristics.

[0008] Preferably, the generated structural consistency feature set F3 includes: Based on the pixel range of the candidate region set in the visible light image data, the corresponding image region is extracted, and the image region is grayscaled. The texture statistics under different directions and different pixel distances are calculated by the gray-level co-occurrence matrix to construct the texture distribution features of the candidate region. Based on the obtained texture distribution features, the image region is color space converted, and the original image is converted from red-green-blue color space to comprehensive chromaticity-luminance color space. The color change amplitude of comprehensive chromaticity component and luminance component in the spatial neighborhood within the candidate region is calculated to obtain color stability features. Based on the texture distribution features and color stability features, an image surface structure description vector is constructed, and the image surface structure description vector is mapped to the spatial shape parameters in the morphological feature set to form a structure-related feature vector. The structural association feature vectors are normalized and combined according to a fixed feature order to generate a set of structural consistency features.

[0009] Preferably, the construction of the fused feature vector includes: Normalize the feature parameters in the set of morphological features, the set of material variation features, and the set of structural consistency features, and construct an initial feature sequence in the order of morphological features, material variation features, and structural consistency features. A cross-modal correlation matrix is ​​constructed based on the initial feature sequence. The correlation coefficients between morphological feature parameters and material change feature parameters, between material change feature parameters and structural consistency feature parameters, and between morphological feature parameters and structural consistency feature parameters are calculated to form a cross-modal feature correlation matrix. The initial feature sequence is weighted and mapped according to the cross-modal feature correlation matrix. By multiplying each feature parameter with its corresponding correlation weight, a cross-modal coupled feature sequence is obtained. The cross-modal coupled feature sequence is subjected to feature reconstruction processing. By calculating the comprehensive response value between each coupled feature and combining them in a fixed order, a fused feature vector is constructed.

[0010] Preferably, the fused feature vector F is input into the waste material identification model for classification and discrimination to obtain candidate identification results, including: The fused feature vector is subjected to feature standardization processing by calculating the difference between each feature value and its average value in the fused feature vector and dividing it by the standard deviation of the corresponding feature value, thereby constructing a standardized fused feature vector; A waste material identification model is constructed based on historical waste material samples. Specifically, the fused feature vectors of multiple labeled categories are used as training samples and input into a multi-layer feedforward neural network for iterative training. The backpropagation algorithm is used to adjust the network weight parameters so that the classification error function is gradually reduced, thereby obtaining the trained waste material identification model. The standardized fusion feature vector is input into the trained waste material identification model, and the classification response value corresponding to each category is obtained through layer-by-layer feature mapping operation; The maximum response category is calculated based on the classification response value corresponding to each category, and the corresponding category is used as the candidate identification result for waste materials.

[0011] Preferably, based on the candidate identification results, a local sensitivity analysis is performed on the fused feature vector to identify the key feature subset that has the greatest impact on the classification results, including: Based on the candidate identification results, the corresponding classification response value is determined, and the classification response value is used as the target value for sensitivity analysis. Each feature parameter in the fused feature vector is perturbed sequentially. A perturbed feature value is generated by adding a preset perturbing amplitude to the original feature parameter, and the fused feature vector containing the perturbed feature value is re-input into the waste material identification model for classification calculation. The influence of each feature parameter on the classification result is determined by calculating the change amplitude of the classification response value before and after perturbing, and a feature sensitivity sequence is constructed based on the influence degree. The feature parameters in the fused feature vector are sorted according to the feature sensitivity sequence, and feature parameters with sensitivity higher than a preset sensitivity threshold are selected to form a key feature subset to characterize the feature set that has the greatest impact on the classification result.

[0012] Preferably, the fused feature vector is reconstructed based on the subset of key features to form an updated fused feature, including: Normalized weights are calculated based on the feature sensitivity values ​​corresponding to each feature parameter in the key feature subset. The weight coefficient of the corresponding feature parameter is obtained by dividing each feature sensitivity value by the sum of all feature sensitivity values ​​in the key feature subset. Each feature parameter in the key feature subset is multiplied by its corresponding weight coefficient to obtain a weighted key feature value. A weighted key feature sequence is then constructed according to the original feature order in the key feature subset. This weighted key feature sequence is mapped back to the corresponding feature position in the fused feature vector, and the corresponding feature value in the original fused feature vector is replaced with the weighted key feature value, while keeping the values ​​of feature parameters not selected in the key feature subset unchanged, thus forming a reconstructed feature sequence. The reconstructed feature sequence is then normalized to ensure that all feature values ​​are distributed within a uniform numerical range, thereby generating an updated fused feature vector.

[0013] This invention also provides a waste material identification system based on multimodal fusion, comprising: The data acquisition module is used to acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S. The morphological feature generation module is used to perform target detection and instance segmentation on the visible light image data I to obtain a candidate region set R of waste materials, and extract the corresponding three-dimensional morphological parameters from the depth structure data D based on the candidate region set R to form a morphological feature set F1. The material spectral analysis module is used to extract the material reflection curve from the near-infrared spectral response data S based on the candidate region R, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2. The structural consistency feature generation module is used to extract texture distribution features and color stability features from the visible light image data I based on the candidate region set R, and combine them with the morphological feature set F1 to generate a structural consistency feature set F3; The cross-modal feature fusion module is used to perform cross-modal correlation mapping on the morphological feature set F1, the material change feature set F2, and the structural consistency feature set F3 to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials. The initial classification and recognition module is used to input the fused feature vector F into the waste material recognition model for classification and discrimination, and to obtain candidate recognition results; The feature reconstruction module is used to perform local sensitivity analysis on the fused feature vector based on the candidate recognition results, identify the key feature subset that has the greatest impact on the classification results, and reconstruct the weights of the fused feature vector based on the key feature subset to form an updated fused feature. The classification and discrimination module is used to input the updated fused features back into the recognition model for discrimination and output the final waste material category recognition result.

[0014] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention constructs a multimodal data fusion recognition framework using visible light image data, depth structure data, and near-infrared spectral response data. During the identification of waste materials, it simultaneously utilizes object appearance information, spatial morphology information, and material spectral information to achieve a collaborative expression of multidimensional features of waste materials. By performing target detection and instance segmentation on visible light image data to determine candidate regions, and combining this with depth structure data to extract three-dimensional morphological parameters and near-infrared spectral response data to calculate the local gradient change characteristics of the material's reflectance curve, the recognition process not only relies on surface visual features but also comprehensively reflects the material properties and structural characteristics of the waste materials. This effectively reduces recognition errors caused by similar appearance, surface contamination, or partial occlusion, improving the accuracy and stability of waste material identification.

[0015] 2. This invention constructs a fused feature vector reflecting the coupling relationship between the microstructure and material properties of waste materials by performing cross-modal correlation mapping on sets of morphological features, material variation features, and structural consistency features. During the identification process, a local sensitivity analysis method is introduced to identify and reconstruct the weights of the key feature subsets in the fused feature vector that have the greatest impact on the classification results. This allows the identification model to highlight key discriminative features and suppress the interference of redundant features. After feature reconstruction, classification is performed again, thereby improving the reliability and accuracy of the classification results. This invention enables more accurate identification of different types of waste materials in complex recycling and sorting scenarios, demonstrating high practical value. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0017] Figure 1 This is a flowchart of the method of the present invention.

[0018] Figure 2 This is a flowchart of the system modules of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1, please refer to Figure 1 As shown in this embodiment, the waste material identification method based on multimodal fusion includes: Acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S.

[0021] In one embodiment of the present invention, multimodal data of the waste materials to be identified in the recycling and sorting scenario are first acquired. The multimodal data includes visible light image data I, depth structure data D, and near-infrared spectral response data S.

[0022] Specifically, a multimodal data acquisition device is installed above the material conveying channel of the recycling and sorting equipment. This device includes a visible light camera, a depth sensor, and a near-infrared spectral sensor. The visible light camera collects surface image information of the waste materials in the visible light band to obtain visible light image data I reflecting the object's color, texture, and appearance. The depth sensor acquires spatial distance information of the waste materials, generating corresponding depth structure data D by measuring the change in distance between the object's surface and the sensor, thus characterizing the three-dimensional morphological structure of the waste materials. The near-infrared spectral sensor collects the reflectance spectrum information of the waste materials in the near-infrared band, thereby obtaining near-infrared spectral response data S that reflects differences in material composition.

[0023] To ensure the correspondence between different modal data, the visible light camera, depth sensor, and near-infrared spectral sensor are time-synchronized during the acquisition process, enabling them to acquire data on the same waste material simultaneously. Simultaneously, by pre-calibrating the spatial relationships between the sensors, the acquired visible light image data (I), depth structure data (D), and near-infrared spectral response data (S) are spatially aligned, ensuring consistency across modal data within a unified coordinate system. This generates multimodal raw data corresponding to the same waste material, providing foundational data for subsequent feature extraction and multimodal fusion recognition.

[0024] It should be noted that the multimodal data acquisition process can be carried out continuously as the waste materials move along the conveyor belt, so as to achieve real-time data acquisition of multiple waste material targets.

[0025] The visible light image data I is subjected to target detection and instance segmentation to obtain a candidate region set R of waste materials. Based on the candidate region set R, the corresponding three-dimensional morphological parameters are extracted from the depth structure data D to form a morphological feature set F1.

[0026] In one embodiment of the present invention, the visible light image data is normalized, and the waste material target in the image is initially located using a target detection model based on a convolutional neural network. Specifically, the acquired visible light image is uniformly adjusted to a fixed resolution size of 640×640 pixels, and the pixel values ​​are linearly normalized to map the pixel value range to between 0 and 1. Subsequently, a target detection model is constructed, which consists of an input layer, a feature extraction layer, a feature fusion layer, and a target prediction layer. The feature extraction layer uses a residual convolution structure to extract image features, the feature fusion layer integrates image semantic information at different levels through multi-scale feature fusion, and the target prediction layer generates candidate target positions on the feature map using a sliding window method and filters candidate targets using an intersection-union ratio (IU / R) calculation method. When the IU / R is greater than a preset threshold of 0.5, the candidate target position is retained, thereby obtaining several target candidate boxes.

[0027] Pixel-level instance segmentation is performed on the target candidate boxes to generate corresponding target segmentation masks, and the candidate region set for waste materials is determined based on the target segmentation masks. Specifically, the corresponding image regions are extracted within the target candidate box range and input into the instance segmentation network for pixel-level classification. The instance segmentation network adopts an encoder-decoder structure, where the encoding part extracts region texture features layer by layer through convolutional layers, and the decoding part gradually restores spatial resolution through upsampling operations, outputting a probability value belonging to the waste material target at each pixel position. Subsequently, the output probability map is binarized according to a probability threshold of 0.6, and pixels with probability values ​​greater than 0.6 are marked as target pixels, thereby generating a target segmentation mask, and the candidate region set for waste materials is determined based on the target segmentation mask.

[0028] The candidate region set is mapped to the corresponding depth structure data. The depth values ​​of each pixel within the candidate region are extracted using pixel coordinate matching, and 3D point cloud data corresponding to the candidate region is constructed. Specifically, under the condition that the visible light image and depth structure data have been spatially calibrated, the depth value of each pixel within the candidate region in the depth structure data is extracted according to the one-to-one correspondence of pixel coordinates. The two-dimensional pixel coordinates are then converted into three-dimensional spatial coordinates using the camera intrinsic parameter matrix. The three-dimensional coordinates are calculated by calculating the position of the spatial point in the three-dimensional coordinate system based on the pixel's horizontal coordinate, vertical coordinate, and corresponding depth value, thereby forming 3D point cloud data composed of multiple spatial points.

[0029] The 3D point cloud data is subjected to surface fitting and spatial distribution statistical analysis to calculate 3D morphological parameters, thereby forming a set of morphological features of waste materials. Specifically, firstly, a local plane fitting method based on the least squares principle is used to fit the surface of the 3D point cloud data. By calculating the covariance matrix of the point set in the neighborhood of each point and solving its eigenvalues, the change of the point cloud surface normal vector is obtained, thereby calculating the surface curvature parameter. Subsequently, the spatial bounding box of the target object is calculated based on the maximum spatial boundary range of the 3D point cloud, and the volume parameter is calculated based on the bounding box size. Furthermore, by statistically analyzing the dispersion of the 3D point cloud in each spatial direction, the standard deviation of the spatial distribution is calculated, thereby obtaining the shape dispersion parameter. Finally, the volume parameter, surface curvature parameter, and shape dispersion parameter are combined to form a set of morphological features of waste materials.

[0030] Based on the candidate region R, extract the material reflection curve from the near-infrared spectral response data S, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2.

[0031] In one embodiment of the present invention, spectral reflectance values ​​within a corresponding wavelength range are extracted based on the pixel positions of the candidate region in the near-infrared spectral response data, and a material reflectance curve of the waste material is constructed according to wavelength order. Specifically, the near-infrared spectral sensor performs spectral sampling at fixed wavelength intervals of 10 nanometers within a preset wavelength range of 900 nm to 1700 nm, thereby forming several wavelength sampling points. Based on the pixel coordinates of the candidate region in the near-infrared spectral response data, the reflectance value corresponding to each pixel in the region at each wavelength sampling point is read one by one, and the average value of the reflectance values ​​of all pixels at the same wavelength sampling point is calculated as the material reflectance intensity at that wavelength position. Subsequently, the material reflectance intensities at each wavelength position are arranged in ascending order of wavelength, thereby forming a material reflectance curve of the waste material, where the horizontal axis represents the wavelength value and the vertical axis represents the reflectance intensity at the corresponding wavelength.

[0032] The material reflection curve is spectrally smoothed by fitting adjacent wavelength sampling points using a sliding window polynomial fitting method to eliminate high-frequency noise and obtain a smoothed material reflection curve. Specifically, the sliding window length is set to 7 consecutive wavelength sampling points. Within each sliding window, a third-order polynomial function is used to perform least-squares fitting on the reflection intensity sequence within the window. The third-order polynomial function is a cubic function of the reflection intensity with respect to the wavelength. The fitting coefficients are solved to minimize the sum of squared fitting errors. Then, the calculation result of the fitting function at the center wavelength of the window is used as the smoothed reflection intensity at that position. The sliding window is gradually moved along the wavelength direction until the smoothing of the entire material reflection curve is completed, thus obtaining the smoothed material reflection curve.

[0033] A fractional derivative spectral analysis model is constructed based on the smoothed material reflection curve. The Grünwald-Letnikov fractional derivative calculation method is used to perform fractional differential operations on the material reflection curve to obtain the local gradient change values ​​at different wavelength positions. Specifically, the smoothed material reflection curve is represented as a discrete function of reflection intensity with respect to wavelength, and the fractional order is set to 0.5, with a wavelength sampling interval of 10 nanometers. According to the definition of the Grünwald-Letnikov fractional derivative, for each wavelength sampling point, the reflection intensity at several preceding wavelength positions is weighted and summed using the fractional difference calculation formula. Each weight coefficient is calculated from fractional binomial coefficients, thereby obtaining the fractional derivative value corresponding to that wavelength position. This fractional derivative value represents the degree of local gradient change at that wavelength position.

[0034] The gradient change intensity parameter and gradient change distribution parameter are calculated based on the local gradient change values ​​corresponding to each wavelength position, and then combined to form a material change feature set. Specifically, firstly, the absolute values ​​of the local gradient change values ​​at all wavelength positions are calculated and averaged to obtain the gradient change intensity parameter, which characterizes the overall change amplitude of the material's reflection curve. Then, the standard deviation of the local gradient change values ​​at each wavelength position is calculated to obtain the gradient change distribution parameter, which characterizes the dispersion of gradient changes at different wavelength positions. Finally, the gradient change intensity parameter and gradient change distribution parameter are combined in a fixed order to form a material change feature set for waste materials, which is used for subsequent multimodal feature fusion and waste material identification.

[0035] Based on the candidate region set R, texture distribution features and color stability features are extracted from the visible light image data I, and a structural consistency feature set F3 is generated by combining the morphological feature set F1.

[0036] In one embodiment of the present invention, image regions corresponding to the candidate region set are extracted from the pixel range in the visible light image data, and the image regions are grayscaled. Texture statistics under different directions and pixel distances are calculated using a gray-level co-occurrence matrix to construct the texture distribution features of the candidate regions. Specifically, the corresponding image regions are extracted according to the pixel coordinate range recorded in the candidate region set, and the three-channel image of the image region, consisting of red, green, and blue components, is converted into a single-channel grayscale image. The grayscale values ​​are calculated by weighting the red component by 0.299, the green component by 0.587, and the blue component by 0.114. Subsequently, a gray-level co-occurrence matrix is ​​constructed on the grayscale image, with 64 grayscale levels and a pixel distance of 1 pixel. The co-occurrence frequency of grayscale values ​​is statistically analyzed in the 0-degree, 45-degree, 90-degree, and 135-degree directions. Based on the obtained gray-level co-occurrence matrix, four texture statistics—energy value, contrast value, correlation value, and homogeneity value—are calculated, and these four statistics are combined in a fixed order to form the texture distribution features of the candidate regions.

[0037] Based on the obtained texture distribution features, the image region is color space transformed, and the color variation amplitudes of the comprehensive chromaticity component and luminance component within the candidate region are calculated in the spatial neighborhood to obtain color stability features. Specifically, the image region is converted from a color space composed of red, green, and blue components to a comprehensive chromaticity-luminance color space, where the comprehensive chromaticity component consists of comprehensive chromaticity component one and comprehensive chromaticity component two, and the luminance component is used to represent the pixel luminance value. Subsequently, a spatial neighborhood with a radius of 3 pixels is established with each pixel as the center, and the differences between comprehensive chromaticity component one, comprehensive chromaticity component two, and luminance component within the neighborhood and the center pixel are calculated. The absolute values ​​of the differences of all pixels are calculated and averaged to obtain the comprehensive chromaticity variation amplitude parameter and the luminance variation amplitude parameter, and the comprehensive chromaticity variation amplitude parameter and the luminance variation amplitude parameter are combined to form the color stability features.

[0038] An image surface structure description vector is constructed based on the texture distribution features and color stability features. This vector is then mapped to spatial shape parameters in the morphological feature set to form a structure-related feature vector. Specifically, the energy, contrast, correlation, and homogeneity values ​​from the texture distribution features are combined with the comprehensive chromaticity variation amplitude and luminance variation amplitude parameters from the color stability features in a fixed order to form the image surface structure description vector. Subsequently, the volume parameters, surface curvature parameters, and shape dispersion parameters from the morphological feature set are read and a correspondence is established according to the parameter order. New feature values ​​are obtained by multiplying each element of the image surface structure description vector with its corresponding spatial shape parameter, thus forming the structure-related feature vector.

[0039] The structural association feature vectors are normalized and combined according to a fixed feature order to generate a structural consistency feature set. Specifically, firstly, each feature value in the structural association feature vector is normalized to its maximum and minimum values. The normalized feature value is calculated by subtracting the minimum value from the feature vector and dividing by the difference between the maximum and minimum values, mapping all feature values ​​to between 0 and 1. Subsequently, the normalized feature values ​​are combined according to a pre-defined feature arrangement order to form a structural consistency feature set, which is used for subsequent multimodal feature fusion processing and waste material category identification.

[0040] The morphological feature set F1, the material variation feature set F2, and the structural consistency feature set F3 are subjected to cross-modal correlation mapping to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials.

[0041] In one embodiment of the present invention, the feature parameters in the morphological feature set, material variation feature set, and structural consistency feature set are subjected to uniform scale normalization, and an initial feature sequence is constructed in the order of morphological features, material variation features, and structural consistency features. Specifically, the feature parameters in each feature set are normalized to their maximum and minimum values. Let a certain feature parameter be a feature value; its normalization calculation method is: subtract the minimum feature value in the corresponding feature set from the feature value, and then divide by the difference between the maximum and minimum feature values ​​in the feature set, so that the normalized feature value is between 0 and 1. After the normalization process is completed, the volume parameters, surface curvature parameters, and shape dispersion parameters in the morphological feature set are arranged in a predetermined order; the gradient change intensity parameters and gradient change distribution parameters in the material variation feature set are arranged in sequence; and the normalized feature values ​​in the structural consistency feature set are arranged in sequence, thereby forming an initial feature sequence by combining the morphological features, material variation features, and structural consistency features in that order.

[0042] A cross-modal correlation matrix is ​​constructed based on the initial feature sequence. This is achieved by calculating the correlation coefficients between morphological feature parameters and material variation feature parameters, between material variation feature parameters and structural consistency feature parameters, and between morphological feature parameters and structural consistency feature parameters. Specifically, assuming the number of feature parameters in the initial feature sequence is n, each feature parameter is denoted as a feature variable. The correlation value is obtained by calculating the Pearson correlation coefficient between any two feature variables. The Pearson correlation coefficient is obtained by calculating the ratio of the product of the covariance and standard deviation of the two feature variables, where the covariance is calculated by summing the products of the differences between each sample feature value and its mean. By calculating the correlation coefficients pairwise for all feature variables, an n x n cross-modal feature correlation matrix is ​​formed, where each element in the matrix represents the degree of correlation between two different modal feature parameters.

[0043] The initial feature sequence is weighted and mapped based on the cross-modal feature correlation matrix. This is achieved by multiplying each feature parameter with its corresponding correlation weight to obtain a cross-modal coupled feature sequence. Specifically, for each feature parameter in the initial feature sequence, all correlation coefficients in the corresponding row of the cross-modal feature correlation matrix are extracted, and the average of these coefficients is calculated to obtain the correlation weight of that feature parameter. Subsequently, each feature parameter in the initial feature sequence is multiplied with its corresponding correlation weight to obtain a cross-modal coupled feature sequence containing multiple coupled feature values.

[0044] The cross-modal coupling feature sequence is reconstructed by calculating the comprehensive response values ​​between each coupling feature and combining them in a fixed order to construct a fusion feature vector reflecting the coupling relationship between the microstructure and material properties of the waste materials. Specifically, the absolute difference between each coupling feature value and other coupling feature values ​​in the cross-modal coupling feature sequence is calculated, and all differences are accumulated to obtain the comprehensive response value of that coupling feature value. Then, all comprehensive response values ​​are combined according to the initial feature sequence to form the final fusion feature vector, which is used in the subsequent waste material category identification process.

[0045] The fused feature vector F is input into the waste material identification model for classification and discrimination to obtain candidate identification results.

[0046] In one embodiment of the present invention, the fused feature vector is subjected to feature standardization to eliminate the influence of differences in the dimensions of different features on the classification results. Specifically, the fused feature vector contains several feature values, and each feature value is denoted as a feature variable. First, the mean and standard deviation of each feature variable in the training sample set are calculated. The mean is obtained by summing the values ​​of the corresponding feature variables in all training samples and dividing by the number of samples. The standard deviation is obtained by averaging the squares of the differences between each feature variable and its mean, and then taking the square root. Subsequently, a standardization operation is performed on each feature variable in the fused feature vector corresponding to the waste material to be identified. By subtracting the corresponding mean from the feature variable and dividing by the corresponding standard deviation, the processed feature variable satisfies a distribution with a mean of zero and a variance of one, thereby forming a standardized fused feature vector.

[0047] A waste material identification model is constructed based on historical waste material samples. Specifically, firstly, multiple waste material samples of known categories are collected, and the fused feature vector corresponding to each sample is calculated according to the aforementioned steps. Each fused feature vector and its corresponding waste material category label are combined to form a training sample dataset. Subsequently, a multi-layer feedforward neural network structure is constructed, where the number of nodes in the input layer is consistent with the number of features in the fused feature vector. The hidden layer is set to a two-layer structure, with the first hidden layer containing 64 neurons and the second hidden layer containing 32 neurons. The activation function of each hidden layer neuron adopts a modified linear function, whose function expression is to output the input value when the input value is greater than zero, and output zero when the input value is less than or equal to zero. The number of neurons in the output layer is the same as the number of waste material categories, and an exponential normalization function is used to calculate the output probability of each category. The training sample dataset is then input into the multilayer feedforward neural network for iterative training. In each training round, the cross-entropy loss value between the predicted class probability and the true class label is first calculated. Then, the gradient of each network weight parameter is calculated through the backpropagation algorithm, and the network weight parameters are updated using the gradient descent method. The learning rate is set to 0.001, and the number of training iterations is set to 200 rounds, thereby obtaining the trained waste material identification model.

[0048] The standardized fused feature vectors are input into the trained waste material identification model for classification. Specifically, the standardized fused feature vectors are sequentially input into the input layer nodes. A weighted summation operation is performed to multiply the feature values ​​by the weight parameters between the input layer and the first hidden layer, and the summation is calculated. Then, a modified linear function is applied to obtain the output value of the first hidden layer. Subsequently, the output value of the first hidden layer is input into the second hidden layer, and the same weighted summation and activation function operation is used to obtain the output value of the second hidden layer. Finally, the output value of the second hidden layer is input into the output layer, and a weighted summation operation combined with an exponential normalization function is used to calculate the classification response value corresponding to each waste material category.

[0049] Candidate identification results for waste materials are determined based on the classification response values ​​corresponding to each category. Specifically, the response values ​​of all categories calculated by the output layer are compared, and the category with the largest value is selected as the predicted category. This category is then used as the candidate identification result for the waste materials, and the response value corresponding to this category is recorded as the classification confidence level for subsequent reliability analysis of the identification results.

[0050] Based on the candidate identification results, a local sensitivity analysis is performed on the fused feature vector to identify the key feature subset that has the greatest impact on the classification results. The weights of the fused feature vector are then reconstructed based on the key feature subset to form the updated fused features.

[0051] In one embodiment of the present invention, a corresponding classification response value is determined based on the candidate identification result, and this classification response value is used as the target value for sensitivity analysis. Specifically, after completing the classification calculation of the waste material identification model, the output layer obtains classification response values ​​corresponding to multiple waste material categories, among which the category with the largest value is determined as the candidate identification result. The classification response value corresponding to the candidate identification result is recorded as the target response value. The target response value represents the confidence level of the current fused feature vector in classifying that category in the waste material identification model, and serves as a reference evaluation index in the subsequent sensitivity analysis process to measure the impact of changes in different feature parameters on the classification result.

[0052] Each feature parameter in the fused feature vector is sequentially perturbed. A perturbed feature value is generated by adding a preset perturbation amplitude to the original feature parameter. The fused feature vector containing the perturbed feature value is then re-inputted into the waste material identification model for classification calculation. Specifically, the fused feature vector contains several feature parameters, each denoted as a feature variable. First, the perturbation amplitude parameter is determined. This parameter is obtained by calculating the standard deviation of the feature variable in the training sample dataset and multiplying it by a perturbation coefficient, which is set to 0.05. Then, each feature variable in the fused feature vector is sequentially perturbed. The original feature variable value is added to the corresponding perturbation amplitude parameter to obtain the perturbed feature value. This perturbed feature value replaces the corresponding feature variable in the fused feature vector, while the remaining feature variables remain unchanged, thus forming the perturbed fused feature vector. The perturbed fused feature vector is then re-inputted into the trained waste material identification model for classification calculation, and the new classification response value is recorded.

[0053] The influence of each feature parameter on the classification result is determined by calculating the change in classification response value before and after perturbation, and a feature sensitivity sequence is constructed based on the degree of influence. Specifically, a perturbation process is performed on each feature variable once to obtain the corresponding perturbation classification response value. Then, the difference between the perturbation classification response value and the target response value is calculated, and the absolute value of the difference is taken to obtain the response change magnitude of that feature variable. The response change magnitude is used to characterize the degree of influence of the feature variable on the classification result. The response change magnitudes corresponding to all feature variables are recorded in the original arrangement order in the fused feature vector to form a feature sensitivity sequence.

[0054] The feature parameters in the fused feature vector are sorted according to the feature sensitivity sequence, and feature parameters with sensitivity higher than a preset sensitivity threshold are selected to form a key feature subset. Specifically, firstly, all sensitivity values ​​in the feature sensitivity sequence are sorted from largest to smallest, and the average sensitivity is calculated as the sensitivity threshold. Then, feature variables with sensitivity values ​​greater than the sensitivity threshold are selected from the sorting results, and these feature variables are combined according to their original order in the fused feature vector to form the key feature subset. The key feature subset is used to represent the set of features that have the greatest impact on the classification result and serves as input data for the subsequent fused feature vector weight reconstruction process.

[0055] In one embodiment of the present invention, normalized weights are calculated based on the feature sensitivity values ​​corresponding to each feature parameter in the key feature subset. Specifically, the key feature subset contains several feature parameters, each corresponding to a feature sensitivity value. All feature sensitivity values ​​in the key feature subset are summed to obtain a total sensitivity. Then, each feature sensitivity value is divided by this total sensitivity to obtain the weight coefficient of the corresponding feature parameter. This calculation method ensures that the sum of all weight coefficients is 1, giving different feature parameters in the key feature subset a clear relative contribution ratio in subsequent calculations.

[0056] Each feature parameter in the key feature subset is multiplied by its corresponding weight coefficient to obtain a weighted key feature value. A weighted key feature sequence is then constructed according to the original feature order within the key feature subset. Specifically, for each feature parameter in the key feature subset, its corresponding weight coefficient is read, and the feature parameter value is multiplied by the corresponding weight coefficient to obtain a weighted key feature value. Subsequently, all weighted key feature values ​​are arranged according to the original order of the key feature subset in the fused feature vector to form a weighted key feature sequence, ensuring that the positional correspondence remains consistent in subsequent feature mapping processes.

[0057] The weighted key feature sequence is mapped back to its corresponding feature position in the fused feature vector, and the corresponding feature value in the original fused feature vector is replaced with the weighted key feature value. Simultaneously, the values ​​of feature parameters not selected in the key feature subset remain unchanged, thus forming a reconstructed feature sequence. Specifically, based on the original index position of each feature parameter in the key feature subset within the fused feature vector, the weighted key feature values ​​from the weighted key feature sequence are written one by one into the corresponding position in the fused feature vector, replacing the original feature values. For feature parameters in the fused feature vector not selected in the key feature subset, their original values ​​remain unchanged. Through the above replacement operation, a new feature arrangement result is obtained, thus forming the reconstructed feature sequence.

[0058] The reconstructed feature sequence is then normalized to ensure all feature values ​​fall within a uniform numerical range, thereby generating an updated fused feature vector. Specifically, maximum and minimum value normalization is performed on all feature values ​​in the reconstructed feature sequence. First, the maximum and minimum feature values ​​are calculated in the reconstructed feature sequence. Then, each feature value is subtracted from the minimum feature value and divided by the difference between the maximum and minimum feature values, ensuring that the normalized feature values ​​are distributed between 0 and 1. After normalization, the updated fused feature vector is obtained and used as input feature data in the subsequent waste material identification process.

[0059] The updated fusion features are then input into the recognition model for discrimination, and the final waste material category recognition result is output.

[0060] In one embodiment of the present invention, the updated fused feature vector is first input into the waste material identification model according to the feature arrangement order required by the input layer, so that each feature parameter in the updated fused feature vector corresponds sequentially to each neuron node of the input layer. Then, a forward propagation calculation process is performed, that is, the feature values ​​of each neuron node in the input layer are multiplied and summed with the weight parameters connected to the neurons in the first hidden layer, thereby obtaining the input value of the first hidden layer neuron. The input value is then substituted into a modified linear function for activation, wherein the input value is output when it is greater than zero, and zero is output when it is less than or equal to zero, thereby obtaining the output value of the first hidden layer.

[0061] The output value of the first hidden layer is then used as the input data for the second hidden layer, and weighted summation and modified linear function activation are performed in the same manner to obtain the output value of the second hidden layer. The output value of the second hidden layer is then input to the output layer neurons. The output layer input value is obtained by multiplying and summing the output value of the second hidden layer with the connection weights of the output layer. An exponential normalization function is then used to normalize the output values ​​of all categories. This exponential normalization function calculates the exponential function of the output value for each category and divides this exponential value by the sum of the exponential values ​​of all categories to obtain the probability value corresponding to each category of waste material.

[0062] Finally, the probability values ​​of each waste material category calculated by the output layer are compared, and the category with the highest probability value is selected as the final identification category. This category is then determined as the final category identification result of the waste material, and the probability value corresponding to this category is recorded as the final identification confidence level, which is used to characterize the reliability of this waste material identification result.

[0063] Example 2, please refer to Figure 2 As shown, the waste material identification system based on multimodal fusion described in this embodiment includes: The data acquisition module is used to acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S. The morphological feature generation module is used to perform target detection and instance segmentation on the visible light image data I to obtain a candidate region set R of waste materials, and extract the corresponding three-dimensional morphological parameters from the depth structure data D based on the candidate region set R to form a morphological feature set F1. The material spectral analysis module is used to extract the material reflection curve from the near-infrared spectral response data S based on the candidate region R, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2. The structural consistency feature generation module is used to extract texture distribution features and color stability features from the visible light image data I based on the candidate region set R, and combine them with the morphological feature set F1 to generate a structural consistency feature set F3; The cross-modal feature fusion module is used to perform cross-modal correlation mapping on the morphological feature set F1, the material change feature set F2, and the structural consistency feature set F3 to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials. The initial classification and recognition module is used to input the fused feature vector F into the waste material recognition model for classification and discrimination, and to obtain candidate recognition results; The feature reconstruction module is used to perform local sensitivity analysis on the fused feature vector based on the candidate recognition results, identify the key feature subset that has the greatest impact on the classification results, and reconstruct the weights of the fused feature vector based on the key feature subset to form an updated fused feature. The classification and discrimination module is used to input the updated fused features back into the recognition model for discrimination and output the final waste material category recognition result.

[0064] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A waste material identification method based on multimodal fusion, characterized in that: include: Acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S; The visible light image data I is subjected to target detection and instance segmentation to obtain a candidate region set R of waste materials. Based on the candidate region set R, the corresponding three-dimensional morphological parameters are extracted from the depth structure data D to form a morphological feature set F1. Based on the candidate region R, extract the material reflection curve from the near-infrared spectral response data S, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2; Based on the candidate region set R, texture distribution features and color stability features are extracted from the visible light image data I, and the structural consistency feature set F3 is generated by combining the morphological feature set F1. The morphological feature set F1, the material variation feature set F2, and the structural consistency feature set F3 are subjected to cross-modal correlation mapping to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials. The fused feature vector F is input into the waste material identification model for classification and discrimination to obtain candidate identification results; Based on the candidate identification results, a local sensitivity analysis is performed on the fused feature vector to identify the key feature subset that has the greatest impact on the classification results. The weights of the fused feature vector are then reconstructed based on the key feature subset to form the updated fused features. The updated fusion features are then input into the recognition model for discrimination, and the final waste material category recognition result is output.

2. The waste material identification method based on multimodal fusion according to claim 1, characterized in that: Target detection and instance segmentation are performed on the visible light image data I, including: The visible light image data is normalized, and the waste material targets in the image are initially located using a target detection model based on a convolutional neural network to obtain several target candidate boxes. Pixel-level instance segmentation is performed on the target candidate boxes to generate corresponding target segmentation masks, and the candidate region set of waste materials is determined based on the target segmentation masks.

3. The waste material identification method based on multimodal fusion according to claim 2, characterized in that: The method for obtaining the morphological feature set F1 is as follows: the candidate region set is mapped to the corresponding depth structure data, the depth value of each pixel in the candidate region is extracted by pixel coordinate matching, and the three-dimensional point cloud data corresponding to the candidate region is constructed. The three-dimensional point cloud data is subjected to surface fitting and spatial distribution statistical analysis to calculate three-dimensional morphological parameters, including volume parameters, surface curvature parameters, and shape dispersion parameters, thereby forming a set of morphological features of waste materials.

4. The waste material identification method based on multimodal fusion according to claim 1, characterized in that: Calculating the local gradient change characteristics of the material reflection curve includes: extracting spectral reflectance values ​​within the corresponding wavelength range based on the pixel positions of the candidate region in the near-infrared spectral response data, and constructing the material reflection curve of the waste material according to wavelength order; performing spectral smoothing on the material reflection curve, and fitting adjacent wavelength sampling points using a sliding window polynomial fitting method to obtain a smoothed material reflection curve; constructing a fractional derivative spectral analysis model based on the smoothed material reflection curve, and performing fractional derivative operations on the material reflection curve using the Glenwald-Letnikov fractional derivative calculation method to obtain local gradient change values ​​at different wavelength positions; calculating gradient change intensity parameters and gradient change distribution parameters based on the local gradient change values ​​corresponding to each wavelength position, and combining the gradient change intensity parameters and gradient change distribution parameters to form a set of material change characteristics.

5. The waste material identification method based on multimodal fusion according to claim 1, characterized in that: Generate a set of structural consistency features F3, including: Based on the pixel range of the candidate region set in the visible light image data, the corresponding image region is extracted, and the image region is grayscaled. The texture statistics under different directions and different pixel distances are calculated by the gray-level co-occurrence matrix to construct the texture distribution features of the candidate region. Based on the obtained texture distribution features, the image region is color space converted, and the original image is converted from red-green-blue color space to comprehensive chromaticity-luminance color space. The color change amplitude of comprehensive chromaticity component and luminance component in the spatial neighborhood within the candidate region is calculated to obtain color stability features. Based on the texture distribution features and color stability features, an image surface structure description vector is constructed, and the image surface structure description vector is mapped to the spatial shape parameters in the morphological feature set to form a structure-related feature vector. The structural association feature vectors are normalized and combined according to a fixed feature order to generate a set of structural consistency features.

6. The waste material identification method based on multimodal fusion according to claim 5, characterized in that: The construction of the fused feature vector includes: Normalize the feature parameters in the set of morphological features, the set of material variation features, and the set of structural consistency features, and construct an initial feature sequence in the order of morphological features, material variation features, and structural consistency features. A cross-modal correlation matrix is ​​constructed based on the initial feature sequence. The correlation coefficients between morphological feature parameters and material change feature parameters, between material change feature parameters and structural consistency feature parameters, and between morphological feature parameters and structural consistency feature parameters are calculated to form a cross-modal feature correlation matrix. The initial feature sequence is weighted and mapped according to the cross-modal feature correlation matrix. By multiplying each feature parameter with its corresponding correlation weight, a cross-modal coupled feature sequence is obtained. The cross-modal coupled feature sequence is subjected to feature reconstruction processing. By calculating the comprehensive response value between each coupled feature and combining them in a fixed order, a fused feature vector is constructed.

7. The waste material identification method based on multimodal fusion according to claim 1, characterized in that: The fused feature vector F is input into the waste material identification model for classification and discrimination to obtain candidate identification results, including: The fused feature vector is subjected to feature standardization processing by calculating the difference between each feature value and its average value in the fused feature vector and dividing it by the standard deviation of the corresponding feature value, thereby constructing a standardized fused feature vector; A waste material identification model is constructed based on historical waste material samples. Specifically, the fused feature vectors of multiple labeled categories are used as training samples and input into a multi-layer feedforward neural network for iterative training. The backpropagation algorithm is used to adjust the network weight parameters so that the classification error function is gradually reduced, thereby obtaining the trained waste material identification model. The standardized fusion feature vector is input into the trained waste material identification model, and the classification response value corresponding to each category is obtained through layer-by-layer feature mapping operation; The maximum response category is calculated based on the classification response value corresponding to each category, and the corresponding category is used as the candidate identification result for waste materials.

8. The waste material identification method based on multimodal fusion according to claim 1, characterized in that: Based on the candidate identification results, a local sensitivity analysis is performed on the fused feature vector to identify the key feature subset that has the greatest impact on the classification results, including: Based on the candidate identification results, the corresponding classification response value is determined, and the classification response value is used as the target value for sensitivity analysis. Each feature parameter in the fused feature vector is perturbed sequentially. A perturbed feature value is generated by adding a preset perturbing amplitude to the original feature parameter, and the fused feature vector containing the perturbed feature value is re-input into the waste material identification model for classification calculation. The influence of each feature parameter on the classification result is determined by calculating the change amplitude of the classification response value before and after perturbing, and a feature sensitivity sequence is constructed based on the influence degree. The feature parameters in the fused feature vector are sorted according to the feature sensitivity sequence, and feature parameters with sensitivity higher than a preset sensitivity threshold are selected to form a key feature subset to characterize the feature set that has the greatest impact on the classification result.

9. The waste material identification method based on multimodal fusion according to claim 8, characterized in that: The fused feature vector is reconstructed based on the aforementioned key feature subset to form updated fused features, including: Normalized weights are calculated based on the feature sensitivity values ​​corresponding to each feature parameter in the key feature subset. The weight coefficient of the corresponding feature parameter is obtained by dividing each feature sensitivity value by the sum of all feature sensitivity values ​​in the key feature subset. Each feature parameter in the key feature subset is multiplied by its corresponding weight coefficient to obtain a weighted key feature value. A weighted key feature sequence is then constructed according to the original feature order in the key feature subset. This weighted key feature sequence is mapped back to the corresponding feature position in the fused feature vector, and the corresponding feature value in the original fused feature vector is replaced with the weighted key feature value, while keeping the values ​​of feature parameters not selected in the key feature subset unchanged, thus forming a reconstructed feature sequence. The reconstructed feature sequence is then normalized to ensure that all feature values ​​are distributed within a uniform numerical range, thereby generating an updated fused feature vector.

10. A waste material identification system based on multimodal fusion, used to implement the waste material identification method based on multimodal fusion as described in any one of claims 1-9, characterized in that: include: The data acquisition module is used to acquire multimodal data of waste materials to be identified in the recycling and sorting scenario, including visible light image data I, depth structure data D, and near-infrared spectral response data S. The morphological feature generation module is used to perform target detection and instance segmentation on the visible light image data I to obtain a candidate region set R of waste materials, and extract the corresponding three-dimensional morphological parameters from the depth structure data D based on the candidate region set R to form a morphological feature set F1. The material spectral analysis module is used to extract the material reflection curve from the near-infrared spectral response data S based on the candidate region R, calculate the local gradient change characteristics of the material reflection curve, and construct the material change feature set F2. The structural consistency feature generation module is used to extract texture distribution features and color stability features from the visible light image data I based on the candidate region set R, and combine them with the morphological feature set F1 to generate a structural consistency feature set F3; The cross-modal feature fusion module is used to perform cross-modal correlation mapping on the morphological feature set F1, the material change feature set F2, and the structural consistency feature set F3 to construct a fusion feature vector that reflects the coupling relationship between the microstructure and material of waste materials. The initial classification and recognition module is used to input the fused feature vector F into the waste material recognition model for classification and discrimination, and to obtain candidate recognition results; The feature reconstruction module is used to perform local sensitivity analysis on the fused feature vector based on the candidate recognition results, identify the key feature subset that has the greatest impact on the classification results, and reconstruct the weights of the fused feature vector based on the key feature subset to form an updated fused feature. The classification and discrimination module is used to input the updated fused features back into the recognition model for discrimination and output the final waste material category recognition result.

Images