A plastic near-infrared spectrum classification method based on kernel density estimation data enhancement

Abstract

The application discloses a plastic near-infrared spectrum classification method based on kernel density estimation data enhancement, and high-fidelity synthesized data conforming to original physical characteristics can be generated from a small amount of original data; the method utilizes a Gaussian kernel function and adaptively determines a kernel bandwidth based on a Silverman empirical rule, adopts a wavelength-by-wavelength dimension modeling mode to skillfully avoid the dimension disaster of high-dimensional spectrum, and makes the synthesized spectrum highly consistent with the original spectrum in terms of characteristic absorption peak shape and noise statistical characteristics; the method effectively solves the extremely uneven sample distribution problem caused by the scarcity of some components in the online identification of waste plastics, significantly enhances the robustness and generalization ability of the identification model, and ensures that the purity of single components after sorting can meet the national standard requirements of high-value utilization.

Description

Technical Field

[0001] This invention relates to the field of near-infrared spectroscopy analysis and pattern recognition, specifically to a technique for data augmentation using kernel density estimation to address the sample imbalance problem in plastic classification. Background Technology

[0002] With the deepening of global industrialization, the production and consumption of plastic products have continued to grow due to their lightweight, low cost, and excellent physical and chemical properties. However, as a typical polymer material, plastics generally have a short service life, with about 40% of plastic products having a service life of only 1 to 2 years. This results in a large amount of plastic solid waste being generated in a short period of time, which can cause serious environmental pollution problems if not properly disposed of. Therefore, the scientific, efficient, and resource-based recycling of waste plastics has become a key issue that urgently needs to be addressed in the fields of environmental protection and circular economy.

[0003] Traditional methods for treating waste plastics mainly include landfill and incineration, but both methods have obvious limitations: landfill not only occupies scarce land resources for a long time, but also poses the risk of leachate pollution of soil and groundwater; incineration releases toxic and harmful gases and greenhouse gases, causing secondary environmental pollution. Therefore, developing high-value utilization technologies that can turn waste plastics into valuable resources is an important way to promote green and low-carbon development and achieve closed-loop material recycling.

[0004] In the recycling process of waste plastics, accurate identification and classification are the foundation and core prerequisite for subsequent high-quality reuse. The national standard "Technical Specification for Waste Plastic Recycling" (GB / T 39171-2020) clearly requires that plastic sorting should follow the principles of stability and no secondary pollution, and that the purity of each individual component after sorting must reach a certain standard before it can be utilized for high value. However, achieving high-purity sorting from mixed waste with complex components places extremely high demands on the robustness of the identification algorithm.

[0005] In recent years, near-infrared spectroscopy has become the mainstream technology in the field of automated sorting due to its advantages such as high efficiency, non-destructive testing, strong online detection capability, and good stability.

[0006] Currently, research in the field of waste plastic identification and classification mainly focuses on plastic identification devices, equipment, and mechanical structures, such as CN201310507509.0, CN201820626577.7, CN207808163U, and a few other methods, such as CN119226901A, a method, device, equipment, and medium for plastic classification based on spectral information. However, these methods do not take into account issues such as rare plastic data. Although these technologies have made some progress under specific laboratory conditions, they are mostly based on ideal datasets with uniform sample distribution, often ignoring the significant differences in the proportion of different components in actual industrial circulation. Specifically, existing technologies lack preprocessing mechanisms for extremely imbalanced data, resulting in extremely poor model recognition capabilities for low-frequency components, making it difficult to meet the stringent national standards for the purity of single components.

[0007] In practical industrial sorting scenarios, the component distribution of plastics to be recycled exhibits significant non-uniformity. Influenced by the proportion of the original consumer market, some rare polymers (such as PVC and MPPO) have an extremely low proportion in the waste stream, resulting in a significant "class imbalance" characteristic in the collected near-infrared spectral samples. Existing algorithm improvements or traditional resampling techniques (such as simple random oversampling) often neglect the physical continuity and statistical distribution of spectral data due to mere mathematical replication, easily inducing model overfitting. Undersampling techniques, on the other hand, can cause the loss of key fingerprint region feature information. Therefore, how to construct data augmentation methods that can effectively expand the feature space, preserve the original physical features, and have statistical consistency for small sample categories to improve the robustness and generalization ability of classification models is a technical challenge that urgently needs to be overcome in the field of near-infrared spectral identification of plastics. Summary of the Invention

[0008] This invention proposes a data augmentation method based on kernel density estimation. This method extracts the statistical distribution features of minority class samples, reconstructs a continuous spectral probability density function in the feature space, and then synthesizes spectral data with high fidelity that conforms to the original physical characteristics. This effectively solves the problem of extremely uneven sample distribution caused by the scarcity of some components in the online identification of waste plastics.

[0009] To achieve the above objectives, the present invention employs the following technical solution: A near-infrared spectral classification method for plastics based on kernel density estimation data augmentation includes the following steps: near-infrared spectral data acquisition, spectral data preprocessing, data augmentation based on kernel density estimation, classification model construction and evaluation, and identification of plastic samples to be tested.

[0010] Step 1, Near-infrared spectral data acquisition, specifically includes the following steps: Step 1.1: Collect the plastic samples to be tested. In this example, four light-colored plastics are used as research objects: acrylonitrile-butadiene-styrene copolymer (ABS), polyvinyl chloride (PVC), modified polyphenylene ether (MPPO), and polypropylene (PP).

[0011] Step 1.2: Use a near-infrared spectrometer to perform spectral scanning in the wavelength range of 900-1700nm; in order to eliminate the influence of sample surface inhomogeneity, each sample is randomly scanned twice in different regions. The instrument automatically outputs the average spectrum of 10 scans for each scan, generating a total of 238 near-infrared spectral data to construct the original spectral library.

[0012] Step 2, spectral data preprocessing, specifically includes the following steps: Step 2.1: The Savitzky-Golay (SG) convolutional smoothing algorithm is used to denoise the original spectrum to filter out instrument electronic noise and random environmental interference. During the processing, an appropriate fitting order and window width are selected based on the characteristics of the spectral curve to make the spectral curve smoother while maintaining the shape of the characteristic peaks.

[0013] Step 2.2: Based on SG smoothing, the baseline of the spectral data is corrected using a linear detrending algorithm to eliminate baseline shifts caused by differences in sample shape, surface roughness, and optical path, thereby obtaining preprocessed standardized spectral data.

[0014] Step 3, data augmentation based on kernel density estimation, specifically includes the following steps: Step 3.1: Statistically analyze the sample size of each category of plastics after preprocessing and identify minority classes with significantly smaller sample sizes. In this embodiment, PVC and MPPO are identified as minority classes, and data augmentation is used to balance the sample size of each category.

[0015] Step 3.2: Model the probability density of minority class samples using kernel density estimation. This invention proposes a data augmentation method based on kernel density estimation to generate realistic synthetic data from a very small amount of original data. Specifically, a one-dimensional KDE model is constructed for each wavelength dimension of the spectrum using a Gaussian kernel function to avoid the curse of dimensionality in high-dimensional modeling.

[0016] Step 4: The KDE modeling process includes the following steps: Step 4.1: Let the original near-infrared spectral dataset be:

[0017] in: ,in It is the first c The original number of samples in the class. m It is a spectral feature dimension.

[0018] Step 4.2: During the KDE modeling process, for each feature dimension... The KDE model is constructed separately because the different wavelength characteristics of the spectral data can be approximated as independent distributions.

[0019] For the c The class of i The probability density function of a feature is estimated as follows:

[0020] in: It's bandwidth. It is a kernel function.

[0021] The kernel function is a key component in kernel density estimation, determining the contribution of each data point to the overall density estimate. Plastic spectral data is continuous data obtained through continuous wavelength sampling, and its spectral curves exhibit a natural symmetrical distribution near characteristic wavelengths. The Gaussian kernel function itself possesses continuous and symmetrical probability distribution characteristics, enabling it to accurately match the distribution patterns of the spectral data and avoid density estimation biases caused by kernel asymmetry. As an infinitely differentiable smooth function, the Gaussian kernel function effectively weakens noise interference in small-sample spectral data when performing density fitting on such data. The resulting synthetic data retains the characteristic trends of the original spectrum while exhibiting good smoothness.

[0022] The formula for calculating the Gaussian kernel function is as follows:

[0023] in Substituting this into the KDE formula, we get:

[0024] Based on the c Class 1 i By fitting the KDE model to all sample values ​​of a feature, the probability density distribution of that feature is obtained. In this process, synthetic data matching the original sample size is generated through Monte Carlo inverse transformation sampling, and finally merged into a complete augmented dataset.

[0025] Besides the kernel function, bandwidth is also a key parameter in kernel density estimation, as it determines the smoothness of the estimation. If the bandwidth is too large, the estimation result will be too smooth, which may mask the local features and details of the spectral data, such as characteristic peaks and valleys.

[0026] If the bandwidth is too small, the estimation results will rely too heavily on small sample spectral data, leading to overfitting and increased sensitivity to noise and outliers during spectral acquisition. Common methods for optimizing bandwidth include cross-validation, Silverman's rule of thumb, and interpolation.

[0027] Considering the continuous nature of plastic spectral data, the characteristics of small sample category distribution, and the suitability of the Gaussian kernel function, and to ensure the efficiency and rationality of bandwidth selection, this study calculates the bandwidth based on the Silverman rule. The calculation process is as follows:

[0028] in: The standard deviation of the data; For sample size Step 5, Classification Model Construction and Evaluation, specifically includes the following steps: Step 5.1: Merge the generated synthetic data with the original preprocessed data to construct a balanced augmented dataset, and randomly divide it into training and test sets at a ratio of 50%.

[0029] Step 5.2: Construct a classification model using random forest; introduce the confusion matrix as an evaluation metric and calculate the accuracy. Recall rate and accuracy ( Accuracy This demonstrates the effectiveness of data augmentation in improving classification performance.

[0030] The relevant evaluation indicators are defined as follows:

[0031] in: Samples correctly identified as belonging to the target category. FN This refers to target category samples that have been misclassified as non-target categories. FP This refers to non-target category samples that have been incorrectly classified as the target category. TN Non-target samples correctly predicted by the model.

[0032] Prediction of plastic samples to be tested: Input the near-infrared spectrum of the plastic to be classified into the trained model, and output the plastic category to achieve automated sorting.

[0033] The present invention has the following beneficial effects: This invention addresses the bottleneck problem of "uneven class distribution" in spectral data during actual industrial sorting of waste plastics. It proposes a data augmentation method based on kernel density estimation, which generates high-fidelity synthetic data that conforms to the original physical characteristics using only a small amount of original data. This method utilizes a Gaussian kernel function and adaptively determines the kernel bandwidth based on the Silverman rule. It cleverly avoids the curse of dimensionality in high-dimensional spectra by employing a wavelength-by-wavelength dimensional modeling approach, ensuring that the synthetic spectrum closely matches the original spectrum in terms of characteristic absorption peak morphology and noise statistics. Experimental results show that this data augmentation strategy not only effectively suppresses model overfitting but also significantly improves the identification of extremely rare components, greatly enhancing the robustness and generalization ability of the identification model. This ensures that the purity of the sorted single components meets the national standards for high-value utilization. Attached Figure Description

[0034] Figure 1 Flowchart for waste plastic identification and analysis; Figure 2 Near-infrared spectral curves of four plastics after SG convolution and detrending treatment; Figure 3 KDE synthesized near-infrared spectral curves; Figure 4 Near-infrared spectral identification confusion matrix of the original plastic; Figure 5 Mixed near-infrared spectroscopy is used to identify confusion matrices; Figure 6 Spatial distribution of the original sample and its synthetic sample in three-dimensional principal component analysis. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings. Various substitutions and modifications made based on ordinary technical knowledge and conventional methods in the art without departing from the above-described technical concept of this invention should be included within the scope of this invention.

[0036] Example: This example uses four light-colored plastics—acrylonitrile-butadiene-styrene copolymer (ABS), polyvinyl chloride (PVC), modified polyphenylene oxide (MPPO), and polypropylene (PP)—as standard experimental materials, totaling 119 samples (50 ABS samples, 25 MPPO samples, 7 PVC samples, and 37 PP samples). The spectral acquisition equipment used is the IAS-5000 portable near-infrared spectrometer based on a digital micromirror array, manufactured by Jiangsu Wuxi Xunjie Guangyuan Technology Co., Ltd.

[0037] Reference Figure 1 A flowchart for near-infrared spectral classification of plastics based on kernel density estimation data, including the following steps: Step 1: The experiment collected the four types of plastic samples mentioned above and used a near-infrared spectrometer to acquire raw spectral data under conditions of wavelength range of 900-1700 nm and resolution of 10 nm. During spectral data collection, each sample was placed as close to the spectrometer as possible and scanned ten times to automatically output the average spectrum. Since different locations of the same sample may differ in shape, surface, and thickness, each sample was randomly scanned twice in different regions. Thus, this embodiment generated a total of 238 near-infrared spectra, establishing the raw near-infrared spectral library of plastic samples for this embodiment.

[0038] Step 2: To eliminate the influence of light scattering and instrument noise on near-infrared spectra, the original spectra were standardized and preprocessed. First, a Savitzky-Golay (SG) convolution smoothing filter was used to smooth and filter out high-frequency random noise. Then, a linear de-trending transform was used for baseline correction to eliminate baseline shifts caused by sample surface roughness and optical path differences, resulting in the preprocessed near-infrared spectral curves of the plastic. The four near-infrared spectral curves of the plastic after SG convolution and de-trending processing are shown below. Figure 2 As shown.

[0039] Step 3: Statistically analyze the original sample size of each category of plastics after preprocessing to identify minority plastics with insufficient sample size. In this embodiment, the proportions of each component are highly imbalanced due to market and other practical scenarios. PVC contains only 7 original samples, while MPPO contains 25 original samples, clearly belonging to the minority class of imbalanced data. Therefore, this invention uses kernel density estimation (KDE) to perform dimension-by-dimensional probability density modeling on the minority class samples and generate synthetic data to balance the data size of each category.

[0040] To address the high-dimensionality of spectral data, for each feature dimension A separate one-dimensional KDE model is constructed to circumvent the curse of dimensionality. The example uses a Gaussian kernel function and adaptively calculates the optimal kernel bandwidth for each dimension based on the Silverman rule of thumb. h。

[0041] After constructing the probability density distribution function, random sampling is performed using the Monte Carlo inverse transform sampling technique, and realistic synthetic spectral data is generated following the objective balance principle. In this embodiment, using the KDE generation strategy, 46 synthetic spectral samples are generated for the very few classes of PVC, 10 synthetic spectral samples are appropriately supplemented for the next few classes of MPPO, while the majority classes of ABS and PP are not enhanced. The near-infrared synthetic spectral curves of the four plastics generated by the kernel density estimation method are shown below. Figure 3As shown (where the green curve represents the synthesized spectrum and the light gray curve represents the smoothed spectrum). To further evaluate the quality of the synthesized data from the perspective of feature spatial distribution, the clustering patterns of the original and synthesized samples in the PCA subspace were analyzed. The three-dimensional principal component analysis (PCA) spatial distribution of the original and synthesized samples is shown in the figure. Figure 6 As shown. By Figure 6 As can be seen, the synthetic data and the corresponding original samples exhibit a highly consistent distribution pattern and tight clustering, with clear class boundaries, demonstrating the fidelity of the synthetic spectrum.

[0042] Step 4: Merge the generated synthetic spectral data with the original preprocessed spectral data to construct a balanced training set. Randomly divide the sample set into a training set (50%) and a test set (50%), and use a random forest classification model to train pattern recognition on the balanced dataset.

[0043] To quantify the classification performance before and after data augmentation, a confusion matrix was introduced to evaluate the classification effect, and quantitative analysis was performed using metrics such as precision, recall, and accuracy. Before adding KDE data augmentation, random forest classification modeling was performed directly using the original spectral data, and its test set confusion matrix is ​​shown below. Figure 4 As shown, the recall rate for very few PVC classes is only 83.3% at this point; however, after adding the synthetic data generated by the KDE strategy of this invention for mixed spectral data classification modeling, its mixed near-infrared spectral recognition confusion matrix is ​​as follows: Figure 5 As shown, the recall rate of very few types of PVC significantly increased to 96.9%, and the overall classification accuracy reached 99.3%, effectively solving the problem of missing rare plastics due to class imbalance in actual sorting scenarios.

[0044] For different rare or small sample plastic components, corresponding near-infrared spectral probability density models can be established, and high-fidelity synthetic spectral data can be generated according to the above-disclosed data augmentation steps, thereby achieving scale balance of the entire category of plastic datasets and significantly improving the global recognition accuracy and generalization performance of the classification model in actual industrial sorting environments.

[0045] All technical contents not described in detail in this invention are publicly known technologies.

[0046] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Finally, it should be noted that the specific embodiments described in this application are merely illustrative examples of the main ideas of the present invention. Those skilled in the art can make various modifications or additions to the described specific embodiments or use similar methods to replace them, but this will not depart from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A near-infrared spectral classification method for plastics based on kernel density estimation data enhancement, characterized in that... Includes the following steps: Step S1: Collect raw near-infrared spectral data of different types of plastic samples to construct the raw dataset; Step S2: Preprocess the original dataset by smoothing and detrending through Savitzky-Golay convolution to eliminate noise and baseline drift. Step S3: Determine the sample size of each plastic category in the original dataset and identify minority class samples whose sample size is lower than a preset threshold; Step S4: Use kernel density estimation to perform non-parametric probability density modeling on the spectral feature distribution of the minority class samples, generate synthetic spectral data, until the sample size of each plastic category reaches the target balance. Step S5: Merge the synthesized spectral data with the preprocessed spectral data to construct a balanced training set, and use the classification tree model algorithm to construct a plastic classification and recognition model to achieve accurate identification of mixed waste plastic components.

2. The near-infrared spectral classification method for plastics based on kernel density estimation data enhancement according to claim 1, characterized in that, In step S2, the preprocessed spectral curves retain the differences in absorption characteristics of different plastics in specific wavelength bands, which are used for differentiation and identification. After Savitzky-Golay convolution smoothing and linear detrending normalization preprocessing, scattering interference caused by physical morphology factors such as plastic sample thickness and surface roughness was eliminated. The four plastics exhibited fundamental differences in the overtone and combination frequency absorption characteristics of their molecular hydrogen-containing groups within the near-infrared spectral fingerprint region of 900–1700 nm. These differences form the cornerstone of classification and identification. First, the spectral curves of ABS show a clear and smooth single-peak structure around 1150nm, 1200nm and 1600nm. Its peak shape and response intensity are stable, which constitute the characteristic fingerprint recognition sites of ABS. Second, MPPO has a strong absorption peak at 1200 nm, and is accompanied by obvious secondary absorption features at 1400 nm and 1500 nm. Its overall baseline shape and trough trend are significantly different from the other three polymer materials. Third, although both PVC and PP show obvious absorption responses around 1200nm, PP exhibits an extremely strong sharp double-peak characteristic structure at this wavelength, while PVC only has a wide characteristic valley in this band. Moreover, at 1400nm, the valley depth and shape of PVC are in obvious contrast to the double-peak shape of PP.

3. The near-infrared spectral classification method for plastics based on kernel density estimation data enhancement according to claim 1, characterized in that, In step S3, the preset threshold is adaptively determined based on the average sample size of all plastic categories in the original dataset, and its calculation formula is as follows: in, The preset threshold; This represents the total number of plastic categories in the dataset; For the first The number of original samples of plastic-like substances; This represents the average sample size for each type of plastic. This is an adjustment coefficient, with a value ranging from 0.5 to 1; if the original sample size of a certain type of plastic... If so, then the category is determined to be a minority class sample.

4. The near-infrared spectral classification method for plastics based on kernel density estimation data enhancement according to claim 1, characterized in that: In step S4, in view of the high-dimensional characteristics of spectral data, a one-dimensional kernel density estimation model is constructed separately for each wavelength dimension under each plastic category to perform dimension-by-dimensional probabilistic modeling, so as to avoid the dimensionality curse in high-dimensional joint modeling.

5. The near-infrared spectral classification method for plastics based on kernel density estimation data enhancement according to claim 1, characterized in that: The kernel bandwidth of the Gaussian kernel function Adaptive calculation is performed based on Silverman's rule of thumb, and the calculation formula is as follows: in: The standard deviation of the data; This represents the number of samples.

6. The near-infrared spectral classification method for plastics based on kernel density estimation data enhancement according to claim 1, characterized in that: In step S4, after constructing the probability density functions for each feature dimension, random sampling is performed from the probability density functions using the Monte Carlo inverse transform sampling technique to generate synthetic spectral data that has statistical consistency with the original minority class samples and retains the original spectral physical characteristics.

Images