Multi-mode plastic film solid waste identification method and system

Abstract

The invention discloses a multi-mode plastic film solid waste identification method and system. The method comprises the following steps: acquiring an RGB-HSI data set; designing a multi-stage feature fusion network of an asymmetric double-trunk structure, wherein the multi-stage feature fusion network comprises two feature extraction branches and a fusion unit; one feature extraction branch carries out feature extraction on the RGB image, the other feature extraction branch carries out deep feature extraction on the HSI image, and a fusion unit carries out multi-stage fusion on feature maps of the two feature extraction branches by using an FPN backbone network; combining a multi-stage feature fusion network and an instance segmentation network Mask R-CNN to construct a model, and loading an RGB-HSI data set for training to obtain a prediction model; and multi-modal plastic film solid waste identification is realized by using the prediction model. According to the RGB and HSI hyperspectral feature fusion method provided by the invention, spectral information is mined from deep features by using the difference of spectral features between different materials, and the defects of RGB images are made up.

Description

Technical Field

[0001] The present invention relates to the technical field of solid waste sorting, and in particular to a multimodal plastic film solid waste identification method and system. Background Art

[0002] With the continuous development of artificial intelligence, intelligent detection models based on deep learning are gradually being applied to the field of solid waste sorting. Deep learning-based object detection models are becoming increasingly mature. These models use neural networks to extract and analyze features from captured images, thereby acquiring deep image information. This model simulates the process by which organisms acquire and recognize external information through vision and the brain. Using the Mask R-CNN series of object detection algorithms on RGB images of plastic film solid waste enables simple location and classification of solid waste. Using pneumatic valves allows for intelligent waste sorting. Plastic film solid waste in everyday life is complex and varies in color, shape, texture, and contour. Furthermore, plastic film comes in a wide variety of materials. Using a single RGB image for these conditions can easily lead to misidentification and missed detection by the object detection algorithm, making it difficult to achieve practical sorting requirements. Using a single HSI image can classify plastic film solid waste by material, but it cannot distinguish between different uses of the same material, for example, it can be confused with high-value plastic waste such as beverage bottles and lunch boxes. Furthermore, it cannot distinguish between different colors of plastic film solid waste, significantly impacting subsequent plastic recycling. Therefore, how to finely sort complex plastic film solid waste according to material and color becomes a difficult problem. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is to provide a multimodal plastic film solid waste identification method, comprising the following steps:

[0005] We obtained hyperspectral datasets and corresponding RGB image sets of various plastic film materials, used the K-means clustering algorithm to filter and crop the hyperspectral datasets to obtain the HSI dataset, and combined the HSI dataset and the corresponding RGB image set to form the RGB-HSI dataset.

[0006] A multi-stage feature fusion network with an asymmetric dual-trunk structure is designed, consisting of two feature extraction branches and a fusion unit. One feature extraction branch extracts features from RGB images, while the other extracts deep features from HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of the feature maps of the two feature extraction branches.

[0007] Combine the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model;

[0008] Multimodal plastic film solid waste identification using predictive models.

[0009] Preferably, the method comprises the following steps:

[0010] Three channels of hyperspectral data are randomly selected to form a pseudo-color image. By obtaining n hyperspectral data points from the annotated area of ​​the pseudo-color image, an original hyperspectral dataset of k*n*h consisting of k categories of hyperspectral data is obtained, where n is the number of pixels in each category and h is the number of bands of the hyperspectral data.

[0011] The K-means clustering algorithm is used to cluster the band vectors of the original hyperspectral dataset, automatically dividing the full-band data of each category into i clusters, finding the optimal cluster center, and using the Euclidean distance to calculate the wavelength closest to the cluster center to obtain the effective wavelength i t indivual;

[0012] The bands corresponding to the effective wavelengths are cropped in the wavelength direction, the effective wavelengths are retained, and other redundant wavelengths are removed to form a new HSI dataset.

[0013] Preferably, the data spectrum segment collected by the hyperspectral data set is the near-infrared spectrum segment, ranging from 900nm to 1700nm, and the number of spectrum segments is 224. The data points here are spatial pixels, and the data are the absorption and reflection values ​​of the experimental sample for the hyperspectral spectrum.

[0014] Preferably, the collected hyperspectral data is subjected to spectral preprocessing to form a hyperspectral data set; the preprocessing refers to performing black and white frame correction, which is expressed as:

[0015]

[0016] Among them, DN is the original spectrum data collected, DN w It is the white frame data collected using the standard calibration white board, DN b is the black frame data collected by the cover camera lens, and r is the reflectivity data.

[0017] Preferably, the wavelength closest to the cluster center is calculated using the Euclidean distance, and the calculated wavelength D is expressed as:

[0018]

[0019] Among them, x i represents the i-th sample, y i represents the coordinates of the center of the i-th cluster.

[0020] Preferably, the clustering is performed on the band vectors of the original hyperspectral data set using the K-means clustering algorithm, comprising the following steps:

[0021] Fix the feature center point k and adjust the category of each sample to reduce the loss J, which is calculated by the loss function and expressed as:

[0022]

[0023] Among them, x i represents the i-th sample, c i is x i The cluster to which it belongs, represents the center point corresponding to the cluster, and M is the total number of samples;

[0024] Fix the category of each sample, adjust the feature center k and continue to converge J;

[0025] The above two steps are repeated alternately until J finally converges, at which point the center point and the sample category division converge simultaneously.

[0026] Preferably, the prediction model includes:

[0027] The RGB feature extraction branch extracts features from the RGB image through a CBL structure and MT_conv convolution connected in sequence; the MT_conv convolution includes four convolution layers; the CBL structure includes a convolution layer, a BN layer, and an activation function Relu;

[0028] An HSI feature extraction branch extracts features from the HSI image through a first CBL structure, a DMP structure, a second CBL structure, and a third CBL structure connected in sequence; the DMP structure includes two maximum pooling layers;

[0029] The fusion unit receives the RGB features output by the RGB feature extraction branch and the features output by the HSI feature extraction branch, fuses them using the FPN network, and outputs the fused features. The HSI features first pass through a DMP structure and a maximum pooling layer in sequence, and then enter the FPN network for fusion.

[0030] The instance segmentation network Mask R-CNN receives fused features and outputs prediction information, which includes the category and coordinate information of the target.

[0031] Preferably, the FPN network is used for fusion, specifically: the RGB features and the HSI features are fused in an add manner, and after the fusion, a 3*3 convolution is used to reduce the depth space redundancy.

[0032] Preferably, the use of the prediction model to achieve multimodal plastic film solid waste identification is specifically as follows: the prediction model is deployed in the industrial computer of the sorting equipment, and the RGB camera and the hyperspectral camera are used to simultaneously collect the materials on the conveyor belt and transmit them to the industrial computer. The industrial computer simultaneously predicts the collected RGB data and HSI data to obtain the type, material and coordinate information of the material, and controls the high-frequency pneumatic valve group to sort the required materials.

[0033] The present invention also provides a multimodal plastic film solid waste identification system, comprising:

[0034] The dataset acquisition module obtains hyperspectral datasets and corresponding RGB image sets of various plastic film materials, filters and crops the hyperspectral data using the K-means clustering algorithm to obtain an HSI dataset, and combines the HSI dataset and the corresponding RGB image set to form an RGB-HSI dataset;

[0035] The model design module designs a multi-stage feature fusion network with an asymmetric dual-trunk structure, which includes two feature extraction branches and a fusion unit. One feature extraction branch extracts features from RGB images, while the other extracts deep features from HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of the feature maps of the two feature extraction branches.

[0036] The model training module combines the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model;

[0037] The solid waste identification module uses a predictive model to achieve multimodal plastic film solid waste identification.

[0038] The present invention has the following beneficial effects:

[0039] (1) Using RGB-HSI synchronized information for fusion detection can complement the inability of RGB information to distinguish materials and HSI information to distinguish colors and uses, and can finely classify materials with similar appearances but different materials, or materials with the same materials but different appearances, by combining RGB color, texture and shape information and spectral material information;

[0040] (2) The original hyperspectral data has a large number of redundant bands, which requires a large amount of calculation when used, has a high data storage cost, and a complex analysis process. The spectral dimensionality reduction method based on K-Means clustering can effectively remove bands that have no or little impact on classification. Since K-Means clustering only needs to calculate the average value of all bands belonging to the cluster when finding the center, it is more suitable for large sample data.

[0041] (3) Asymmetric dual-backbone target detection model, that is, the RGB image uses a general feature extraction network as the feature extraction module, and the HSI image is designed with a spectral deep convolution algorithm for feature extraction; and the fusion is performed in the FPN structure. The fused feature map uses one-dimensional convolution to remove the fused redundant information, and finally the category and coordinate information of the target object are obtained through the Mask R-CNN algorithm.

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A diagram showing the steps of a method according to an embodiment of the present invention;

[0044] Figure 2 A spectral point dataset generated when selecting hyperspectral feature bands according to an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of matching an RGB-HSI dataset formed by combining an HSI dataset and a corresponding RGB image set according to an embodiment of the present invention;

[0046] Figure 4 This is a diagram of the prediction model network architecture of an embodiment of the present invention;

[0047] Figure 5 Schematic diagram of an FPN network according to an embodiment of the present invention;

[0048] Figure 6 It is a synchronous acquisition platform for RGB area array cameras and hyperspectral cameras;

[0049] Figure 7 2 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0050] This paper provides a method for fusing RGB and HSI hyperspectral features, involving the selection of hyperspectral feature bands and an RGB-HSI image fusion model. This model exploits the differences in spectral characteristics between different materials to mine spectral information from deep features, thus overcoming the shortcomings of RGB images.

[0051] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0052] S101, obtaining hyperspectral datasets and corresponding RGB image sets of various types of plastic film materials, filtering and cropping the hyperspectral datasets using a K-means clustering algorithm to obtain an HSI dataset, and combining the HSI dataset and the corresponding RGB image sets to form an RGB-HSI dataset;

[0053] S102: Design a multi-stage feature fusion network with an asymmetric dual-backbone structure, including two feature extraction branches and a fusion unit. One feature extraction branch performs feature extraction on RGB images, while the other performs deep feature extraction on HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of the feature maps of the two feature extraction branches.

[0054] S103, combining the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model;

[0055] S104, using the prediction model to achieve multimodal plastic film solid waste identification.

[0056] Specifically, the hyperspectral data set collected in S101 is the near-infrared spectrum, ranging from 900nm to 1700nm, with a spectral range of 224. The data points here are spatial pixels, and the data are the absorption and reflection values ​​of the experimental samples for the hyperspectral. In order to ensure the robustness of the model, spectral preprocessing is required, that is, black and white frame correction is performed. DN is the collected raw spectral data, DN w It is the white frame data collected using the standard calibration white board, DN b is the black frame data collected by the cover camera lens, and r is the reflectivity data correction formula as follows:

[0057]

[0058] For details, see Figure 2 As shown, in S101, the K-means clustering algorithm is used to screen and trim the hyperspectral dataset to obtain the HSI dataset, including the following steps:

[0059] Three channels of hyperspectral data are randomly selected to form a pseudo-color image. By obtaining n hyperspectral data points from the annotated area of ​​the pseudo-color image, an original hyperspectral dataset of k*n*h consisting of k categories of hyperspectral data is obtained, where n is the number of pixels in each category and h is the number of bands of the hyperspectral data. Figure 2 As shown;

[0060] The K-means clustering algorithm is used to cluster the band vectors of the original hyperspectral dataset, automatically dividing the full-band data of each category into i clusters, finding the optimal cluster center, and using the Euclidean distance to calculate the wavelength closest to the cluster center to obtain the effective wavelength i t indivual;

[0061] The bands corresponding to the effective wavelengths are cropped in the wavelength direction, the effective wavelengths are retained, and other redundant wavelengths are removed to form a new HSI dataset.

[0062] Specifically, the Euclidean distance is used to calculate the wavelength closest to the cluster center, and the distance is expressed as:

[0063]

[0064] Specifically, the K-Means clustering method first fixes the feature center k, adjusts the category of each sample to reduce the loss J, then fixes the category of each sample, adjusts the feature center k, and continues to converge J. The above two steps are repeated alternately until J finally converges. At this time, the center point and the sample category classification converge simultaneously. The loss function used by the K-Means clustering method is expressed as:

[0065]

[0066] Among them, x i represents the i-th sample, c i is x i The cluster to which it belongs, represents the center point of the cluster, and M is the total number of samples.

[0067] Specifically, the HSI dataset and the corresponding RGB image set are combined to form the RGB-HSI dataset. It is necessary to match the HSI data and the RGB image. Figure 3 shown.

[0068] For details, see Figure 4 As shown, in S102, the feature extraction network that inputs RGB data alone is modified to a network that can simultaneously input RGB and HSI data. The RGB image features and HSI image features are extracted synchronously by using an asymmetric feature extraction network. In this embodiment, the RGB image uses the ResNet-50 network for feature extraction. The RGB feature map and the HSI feature map are fused in an "add" manner within the FPN network, and a 3*3 convolution is used after the feature addition to reduce the depth space redundancy. The multi-stage feature fusion module retains the rich spatial shape, texture and color gamut information of the RGB image, while adding the rich deep information of the HSI image.

[0069] Specifically, S103 constructs a model based on a multi-stage feature fusion network through a Pytorch framework, and loads an RGB-HSI dataset for training to obtain a prediction model.

[0070] The prediction model includes a multi-stage feature fusion network, an instance segmentation algorithm, and a decision fusion block. The fused features obtained by the multi-stage feature fusion network are input into the instance segmentation algorithm. The results obtained by the detection head and the segmentation head are input into the decision fusion block. The spectral values ​​of the corresponding areas of the spectral image are obtained through the mask of the RGB image. After the spectral values ​​are standardized, the spectral values ​​of each point in the mask are classified using a support vector machine (SVM). The proportion of each category is counted, and the obtained results are fused with the category information and confidence of the instance segmentation algorithm. The fusion process is expressed as follows:

[0071] W=MAX(k1,k2,...,k n );

[0072]

[0073] Among them, k1, k2, …, k n is the proportion of class 1 to class n in an object, where n is the number of categories in the data, W is the highest content proportion among the three categories, K is the threshold for setting each content, Category is the result of the Mask R-CNN algorithm, and C HSI The SVM points are classified into the category with the highest content. Decision fusion outputs the final coordinates, category, and confidence information.

[0074] For details, see Figure 5 Figure 2 shows a schematic diagram of the FPN network of an embodiment of the present invention; the feature fusion module receives the C1-C4 feature maps from the RGB feature extraction module and the three HSI feature maps SF1, SF2, and SF3 from HDConv. The FPN network is used as the fusion carrier, and the fusion structure in FPN is shown in the figure. The RGB feature map and the HSI feature map are fused in an "add" manner within the FPN network. After the feature addition, a 3*3 convolution is used to reduce the depth space redundancy. The multi-stage feature fusion module retains the rich spatial shape, texture, and color gamut information of the RGB image, while adding the rich deep-level information of the HSI image. The input is the feature maps of four different scales of the RGB image. The RGB features and HSI features are added and input into the convolution, and the fused feature map is output to the detection network of Mask R-CNN.

[0075] For details, see Figure 6 As shown, S104 deploys the fusion model trained in step 5 to the industrial computer of the sorting equipment, uses the RGB camera and the hyperspectral camera to simultaneously collect the materials on the conveyor belt and transmit them to the industrial computer. The industrial computer simultaneously predicts the RGB and HSI data to obtain the type, material and coordinate information of the material, and controls the high-frequency pneumatic valve group to sort the required materials.

[0076] For details, see Figure 7 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0077] The data set acquisition module 701 acquires hyperspectral data sets and corresponding RGB image sets of various types of plastic film materials, filters and crops the hyperspectral data using the K-means clustering algorithm to obtain an HSI data set, and combines the HSI data set and the corresponding RGB image set to form an RGB-HSI data set;

[0078] Model design module 702 designs a multi-stage feature fusion network with an asymmetric dual-backbone structure, including two feature extraction branches and a fusion unit. One feature extraction branch performs feature extraction on RGB images, while the other feature extraction branch performs deep feature extraction on HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of feature maps from the two feature extraction branches.

[0079] The model training module 703 combines the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model;

[0080] The solid waste identification module 704 uses a prediction model to implement multimodal plastic film solid waste identification.

[0081] It can be seen that the present invention can effectively solve the characteristics of different shapes, colors and materials of plastic film solid waste by combining RGB image features with HSI image features. The selection of feature bands can reduce data redundancy and lower computing costs. The use of texture and color information of RGB images combined with the high-dimensional material information of HSI can improve detection accuracy and enhance generalization performance.

[0082] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal plastic film solid waste identification method, characterized in that: The following steps are involved: Obtain hyperspectral datasets and corresponding RGB image sets of various plastic film materials, use K-means clustering algorithm to filter and crop the hyperspectral dataset to obtain HSI dataset, and combine the HSI dataset and corresponding RGB image set to form RGB-HSI dataset; Design a multi-stage feature fusion network with an asymmetric dual-trunk structure, including two feature extraction branches and a fusion unit; One feature extraction branch extracts features from RGB images, and the other feature extraction branch extracts deep features from HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of the feature maps of the two feature extraction branches. Combine the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model; Multimodal plastic film solid waste identification using predictive models.

2. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The method of using the K-means clustering algorithm to screen and trim the hyperspectral data set to obtain the HSI data set includes the following steps: The hyperspectral data of three channels are randomly selected to form a pseudo-color image. By obtaining n hyperspectral data points from the annotated area of ​​the pseudo-color image, an original hyperspectral data set of k*n*h consisting of k categories of hyperspectral data is obtained, where n is the number of pixels in each category and h is the number of bands of the hyperspectral data. The K-means clustering algorithm is used to cluster the band vectors of the original hyperspectral data set, automatically dividing the full-band data of each category into i clusters, finding the optimal cluster center, and using the Euclidean distance to calculate the wavelength closest to the cluster center to obtain the effective wavelength i t indivual; The bands corresponding to the effective wavelengths are clipped in the wavelength direction, the effective wavelengths are retained, and other redundant wavelengths are removed to form a new HSI data set.

3. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The data spectrum segment collected by the hyperspectral data set is the near-infrared spectrum segment, ranging from 900nm to 1700nm, and the number of spectrum segments is 224. The data points here are spatial pixels, and the data are the absorption and reflection values ​​of the experimental samples for the hyperspectral spectrum.

4. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The collected hyperspectral data is subjected to spectral preprocessing to form a hyperspectral data set; the preprocessing refers to performing black and white frame correction, which is expressed as: Among them, DN is the original spectral data collected, DN w It is the white frame data collected using the standard calibration white board, DN b is the black frame data collected by the cover camera lens, and r is the reflectivity data.

5. The multimodal plastic film solid waste identification method according to claim 2, characterized in that: The Euclidean distance is used to calculate the wavelength closest to the cluster center. The calculated wavelength D is expressed as: Among them, x i represents the i-th sample, y i represents the coordinates of the center of the i-th cluster.

6. The multimodal plastic film solid waste identification method according to claim 2, characterized in that: The K-means clustering algorithm is used to cluster the band vectors of the original hyperspectral dataset. The following steps are involved: Fix the feature center point k and adjust the category of each sample to reduce the loss J, which is calculated by the loss function and expressed as: Among them, x i represents the i-th sample, c i is x i The cluster to which it belongs, represents the center point corresponding to the cluster, and M is the total number of samples; Fix the category of each sample, adjust the feature center k and continue to converge J; The above two steps are repeated alternately, so that J finally converges, at which point the center point and the sample category division converge simultaneously.

7. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The prediction model includes: The RGB feature extraction branch extracts features from the RGB image through a CBL structure and MT_conv convolution connected in sequence; the MT_conv convolution includes four convolution layers; the CBL structure includes a convolution layer, a BN layer and an activation function Relu; An HSI feature extraction branch extracts features from an HSI image by sequentially connecting a first CBL structure, a DMP structure, a second CBL structure, and a third CBL structure; the DMP structure includes two maximum pooling layers; The fusion unit receives the RGB features output by the RGB feature extraction branch and the features output by the HSI feature extraction branch, fuses them using the FPN network, and outputs the fused features; the HSI features first pass through a DMP structure and a maximum pooling layer in sequence, and then enter the FPN network for fusion; The instance segmentation network Mask R-CNN receives fused features and outputs prediction information, where the prediction information includes the category and coordinate information of the target.

8. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The FPN network is used for fusion, specifically: RGB features and HSI features are fused in an add manner, and after fusion, 3*3 convolution is used to reduce deep spatial redundancy.

9. The multimodal plastic film solid waste identification method according to claim 1, characterized in that: The method of using a prediction model to realize multimodal plastic film solid waste identification is specifically as follows: the prediction model is deployed in the industrial computer of the sorting equipment, and the materials on the conveyor belt are collected simultaneously using an RGB camera and a hyperspectral camera and transmitted to the industrial computer. The industrial computer simultaneously predicts the collected RGB data and HSI data to obtain the type, material and coordinate information of the material, and controls the high-frequency pneumatic valve group to sort the required materials.

10. A multimodal plastic film solid waste identification system, characterized in that: include: The data set acquisition module acquires the hyperspectral data sets and corresponding RGB image sets of various plastic film materials, and uses the K-means clustering algorithm to filter and crop the hyperspectral data to obtain the HSI data set, and combines the HSI data set and the corresponding RGB image set to form the RGB-HSI data set; Model design module, designs a multi-stage feature fusion network with an asymmetric dual-trunk structure, including two feature extraction branches and a fusion unit; One feature extraction branch extracts features from RGB images, and the other feature extraction branch extracts deep features from HSI images. The fusion unit uses the FPN backbone network to perform multi-stage fusion of the feature maps of the two feature extraction branches. The model training module combines the multi-stage feature fusion network and the instance segmentation network Mask R-CNN to build a model and load the RGB-HSI dataset for training to obtain a prediction model; The solid waste identification module uses a predictive model to achieve multimodal plastic film solid waste identification.

Images