A solid waste sorting method and system based on RGB-MSI feature fusion

Abstract

The application discloses a kind of solid waste sorting method and system based on RGB-MSI feature fusion, it is related to spectral feature analysis and depth learning model optimization field, method includes the following steps: S1, obtain hyperspectral dataset;S2, obtain multispectral feature band;S3, obtain multispectral dataset;S4, construct double main stem network YOLO model;S5, obtain RGB-MSI prediction model;S6, solid waste detection sorting.The application uses RGB and MSI fusion idea, makes full use of RGB texture color information and spectral material information, realizes more accurate solid waste identification;The collected hyperspectral data is removed redundant information using improved AP clustering algorithm to adaptively find the best spectral feature band;Double main stem network model uses mid-term feature fusion mode, effectively combines RGB image feature and multispectral feature together, increases the detection accuracy of model, and the fusion method is not only suitable for target detection model, but also can be used in instance segmentation model.

Description

Technical Field

[0001] This invention relates to the fields of spectral feature analysis and deep learning model optimization, and in particular to a solid waste sorting method and system based on RGB-MSI feature fusion. Background Technology

[0002] With the continuous development of artificial intelligence, deep learning-based intelligent detection models are increasingly being applied in industrial fields. Deep learning-based computer vision technology is maturing. Computer vision is a technology that simulates biological vision. Its main task is to extract features from real images or videos and perform related processing to obtain effective information; that is, to simulate the process by which human vision and the brain capture external information. Currently, mainstream image recognition models can be mainly divided into: semantic segmentation, object detection, and instance segmentation.

[0003] Using YOLO's object detection models to detect RGB images of solid waste enables the location and classification of solid waste, and, in conjunction with actuators, allows for intelligent sorting. However, in reality, many solid wastes exhibit striking similarities in color, outline, and texture. In such cases, relying solely on RGB image information can easily lead to misidentification or non-identification by the object detection model. The low accuracy of the object detection model fails to meet the requirements of practical engineering applications. While significantly expanding the dataset can improve model accuracy to a limited extent, the diverse types of solid waste and the increased cost of large datasets do not fundamentally and substantially improve detection accuracy. Therefore, improving the accuracy of object detection models remains a challenging problem. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art and provide an RGB-MSI feature fusion method, which ultimately improves the sorting accuracy of solid waste by selecting spectral feature bands.

[0005] The technical solution adopted by this invention to solve its technical problem is: to provide a solid waste sorting method based on RGB-MSI feature fusion, comprising the following steps: A solid waste sorting method based on RGB-MSI feature fusion, characterized in that it comprises the following steps:

[0006] S101, the step of acquiring the dataset: use a hyperspectral camera and a color camera to acquire hyperspectral images and RGB images of different types of solid waste samples respectively. For each hyperspectral image, N hyperspectral data points are randomly selected. Each data point has M bands. The data points form an N*M hyperspectral dataset. The RGB images are used as the solid waste RGB dataset.

[0007] S102, the step of obtaining multispectral feature bands: transpose the hyperspectral dataset to obtain an M*N transposed hyperspectral dataset, put the transposed hyperspectral dataset into the improved AP clustering algorithm, and adaptively find the optimal cluster center; analyze the band set K0 of the optimal cluster center using the PCA method, and obtain the spectral bands corresponding to the first K maximum values ​​of the PCA first principal component loading curve as multispectral feature bands.

[0008] S103, the step of obtaining a multispectral dataset: based on the multispectral feature bands, the collected hyperspectral image of solid waste is cropped, and the image channels where the multispectral feature bands are located are cropped to form a pseudo-color image with K channels, which is used as a multispectral dataset.

[0009] S104, the step of constructing a dual-backbone YOLO model, adopts an asymmetric dual-backbone network structure to build a YOLO model including two backbone networks; one backbone network is used to extract features from the solid waste RGB dataset to obtain RGB features; the other backbone network is used to extract features from the multispectral dataset to obtain MSI features; finally, the features output by the two backbone networks are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and solid waste coordinates;

[0010] S105, the step of obtaining the RGB-MSI prediction model involves inputting the multispectral dataset and the corresponding solid waste RGB dataset into a dual-backbone YOLO model. The solid waste RGB dataset is used to train one of the backbone networks to extract features from the solid waste RGB dataset, obtaining RGB features; the multispectral dataset is used to train the other backbone network to extract features from the multispectral dataset, obtaining MSI features; finally, the obtained RGB features and MSI features are fused, and the solid waste category is predicted based on the fused features. The output includes the solid waste coordinates and solid waste category; the trained dual-backbone YOLO model is then obtained as the RGB-MSI prediction model.

[0011] S106, Solid waste detection and sorting step: Obtain solid waste RGB images by taking pictures of solid waste with an on-site RGB camera, and obtain solid waste multispectral images by taking pictures of solid waste with a multispectral camera. Input the obtained solid waste RGB images and multispectral images into the RGB-MSI prediction model simultaneously, and obtain solid waste coordinates and solid waste categories to achieve solid waste sorting.

[0012] Preferably, the hyperspectral data points selected in S101 are located in the near-infrared spectral region with a wavelength of 900nm to 1700nm.

[0013] Preferably, in step S101, after preprocessing the hyperspectral data points, a hyperspectral dataset is formed; the hyperspectral data points have high-dimensional features, namely, the reflectance of the near-infrared spectrum; the preprocessing involves black-and-white frame correction of the reflectance, and the correction formula is as follows:

[0014]

[0015] Where r represents the normalized spectral reflectance, and DN is the raw spectral data collected. w It is white frame data acquired using a standard calibration whiteboard, DN b It is black frame data that is obscured by the lens of a hyperspectral camera.

[0016] Preferably, the improved AP clustering algorithm in S102 uses the Pearson correlation coefficient to construct a similarity matrix, performs pairwise Pearson correlation coefficient analysis on all bands, and finally obtains an M*M similarity matrix, which represents the linear relationship between different bands; the Pearson correlation coefficient... Where X and Y represent the two variables for which the Pearson correlation coefficient is to be calculated, and σ X σ represents the sample standard deviation of X. Y Y represents the sample standard deviation, and cov represents the covariance between the two variables.

[0017] Preferably, in the improved AP clustering algorithm in S102, the Preference parameter uses the information entropy formula of Shannon's theorem, sorting the information entropy of the spectral bands from largest to smallest, with the top N information entropies sorted... 信 The weights of the spectral bands are set as the first weight, and the weights of the remaining spectral bands are set as the second weight, where the first weight is greater than the second weight; the weights of the spectral bands form a 1*M preference array, representing the preference of different bands to become cluster centers.

[0018] Preferably, in S102, the second weight is taken as the median of the similarity matrix.

[0019] Preferably, in S102, the central band selected by the AP clustering is analyzed using PCA, the first principal component of PCA is selected, and the feature vector of the first principal component is output. The feature vector corresponds to the weight of each band, and the top K bands with the largest weights are selected as the final multispectral feature bands.

[0020] Preferably, in S104, the feature fusion of the outputs of the two backbone networks specifically involves fusing RGB features and MSI features through a concat operation at the rear of the dual backbone networks.

[0021] Preferably, in S105, the training process requires manual or semi-automatic labeling of the dataset, and the multispectral dataset and the RGB dataset share a single label set.

[0022] This invention also provides a solid waste sorting system based on RGB-MSI feature fusion, comprising the following modules:

[0023] The data acquisition module is used to acquire datasets. It uses a hyperspectral camera and a color camera to acquire hyperspectral images and RGB images of different types of solid waste samples, respectively. N hyperspectral data points are randomly selected from each hyperspectral image. Each data point has M bands, and the data points form an N*M hyperspectral dataset. The RGB images serve as the solid waste RGB dataset.

[0024] The band selection module is used to acquire multispectral feature bands. It transposes the hyperspectral dataset to obtain an M*N transposed hyperspectral dataset, and puts the transposed hyperspectral dataset into the improved AP clustering algorithm to adaptively find the optimal cluster center. The band set K0 of the optimal cluster center is analyzed by PCA method to obtain the spectral bands corresponding to the first K maximum values ​​of the first principal component loading curve of PCA, which are used as multispectral feature bands.

[0025] The dataset construction module is used to acquire a multispectral dataset. Based on the spectral feature bands, the acquired hyperspectral images of solid waste are cropped, and the image channels where the multispectral feature bands are located are cropped to form a pseudo-color image with K channels, which serves as the multispectral dataset.

[0026] The model building module is used to construct a dual-backbone YOLO model. An asymmetric dual-backbone network structure is used to build the YOLO model. One backbone network extracts features from the solid waste RGB dataset to obtain RGB features; the other backbone network extracts features from the multispectral dataset to obtain MSI features. Finally, the features output by the two backbone networks are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and solid waste coordinates.

[0027] The training module is used to obtain the RGB-MSI prediction model. It inputs the multispectral dataset and the corresponding solid waste RGB dataset into a dual-backbone YOLO model. The solid waste RGB dataset is used to train one backbone network to extract features from the solid waste RGB dataset, obtaining RGB features. The multispectral dataset is used to train the other backbone network to extract features from the multispectral dataset, obtaining MSI features. Finally, the obtained RGB and MSI features are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and coordinates. The trained dual-backbone YOLO model is then used as the RGB-MSI prediction model.

[0028] The analysis module is used for solid waste detection and sorting. It captures solid waste images using both an on-site RGB camera and a multispectral camera. The two sets of images are simultaneously input into the RGB-MSI prediction model, which outputs the coordinates and category of the solid waste, thus achieving solid waste sorting.

[0029] The present invention has the following beneficial effects:

[0030] 1. Adopting the concept of RGB and MSI fusion, since RGB information cannot distinguish objects of different materials, while spectral information comes from the different molecular structures of different objects, it can effectively distinguish objects that are similar in appearance but different in material; adopting a feature fusion concept, making full use of RGB texture color information, combined with spectral material information, to achieve more accurate identification of solid waste.

[0031] 2. Since hyperspectral image information contains a lot of redundancy and the cost of hyperspectral data acquisition is also high, this invention uses an improved AP clustering adaptive method to find the best spectral feature bands that do not contain redundant information.

[0032] 3. RGB images and multispectral images have a one-to-one correspondence, so a common label can be used, eliminating the need for additional annotation work.

[0033] 4. The dual-backbone object detection model provides additional input for the model's prediction. It effectively combines RGB image features and multispectral features through mid-term feature fusion, significantly increasing the model's detection accuracy. This fusion method is not only applicable to object detection models but can also be used in instance segmentation models.

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Attached Figure Description

[0035] Figure 1 This is a diagram illustrating the method steps of an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram illustrating the selection of a multispectral dataset in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram illustrating the selection of band features in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of the dual-backbone YOLO model according to an embodiment of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating the training of a dual-backbone YOLO model according to an embodiment of the present invention;

[0040] Figure 6This is a schematic diagram illustrating the detection of on-site solid waste using an RGB-MSI prediction model according to an embodiment of the present invention;

[0041] Figure 7 This is a schematic diagram of the data acquisition platform for detecting on-site solid waste using the RGB-MSI prediction model, as described in an embodiment of the present invention.

[0042] Figure 8 This is a system structure diagram of an embodiment of the present invention. Detailed Implementation

[0043] See Figure 1 The diagram shown illustrates the method steps of an embodiment of the present invention, including:

[0044] S101, the step of acquiring the dataset: use a hyperspectral camera and a color camera to acquire hyperspectral images and RGB images of different types of solid waste samples respectively. For each hyperspectral image, N hyperspectral data points are randomly selected. Each data point has M bands. The data points form an N*M hyperspectral dataset. The RGB images are used as the solid waste RGB dataset.

[0045] S102, the step of obtaining multispectral feature bands: transpose the hyperspectral dataset to obtain an M*N transposed hyperspectral dataset, put the transposed hyperspectral dataset into the improved AP clustering algorithm, and adaptively find the optimal cluster center; analyze the band set K0 of the optimal cluster center using the PCA method, and obtain the spectral bands corresponding to the first K maximum values ​​of the PCA first principal component loading curve as multispectral feature bands.

[0046] S103, the step of obtaining a multispectral dataset: based on the multispectral feature bands, the collected hyperspectral image of solid waste is cropped, and the image channels where the multispectral feature bands are located are cropped to form a pseudo-color image with K channels, which is used as a multispectral dataset.

[0047] S104, the step of constructing a dual-backbone YOLO model, adopts an asymmetric dual-backbone network structure to build a YOLO model including two backbone networks; one backbone network is used to extract features from the solid waste RGB dataset to obtain RGB features; the other backbone network is used to extract features from the multispectral dataset to obtain MSI features; finally, the features output by the two backbone networks are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and solid waste coordinates;

[0048] S105, the step of obtaining the RGB-MSI prediction model involves inputting the multispectral dataset and the corresponding solid waste RGB dataset into a dual-backbone YOLO model. The solid waste RGB dataset is used to train one of the backbone networks to extract features from the solid waste RGB dataset, obtaining RGB features; the multispectral dataset is used to train the other backbone network to extract features from the multispectral dataset, obtaining MSI features; finally, the obtained RGB features and MSI features are fused, and the solid waste category is predicted based on the fused features. The output includes the solid waste coordinates and solid waste category; the trained dual-backbone YOLO model is then obtained as the RGB-MSI prediction model.

[0049] S106, Solid waste detection and sorting step: Obtain solid waste RGB images by taking pictures of solid waste with an on-site RGB camera, and obtain solid waste multispectral images by taking pictures of solid waste with a multispectral camera. Input the obtained solid waste RGB images and multispectral images into the RGB-MSI prediction model simultaneously, and obtain solid waste coordinates and solid waste categories to achieve solid waste sorting.

[0050] For details, see Figure 2 The diagram shows a schematic of selecting a multispectral dataset in an embodiment of the present invention. To avoid robustness, N data points are selected for different individuals of each category. The data points selected for solid waste samples of each category are consistent. For example, 1500 data points are selected for different individuals of category one, and similarly, 1500 data points are selected for categories two, three, or more.

[0051] For details, see Figure 3 The diagram illustrates the selection of band features in an embodiment of the present invention. A hyperspectral camera acquires a hyperspectral image with height h, width w, and number of channels M. Multiple data points are selected from this image, with each data point having band numbers M1, M2, M3, ...

[0052] First, in the first stage of preprocessing, modified AP clustering is used to cluster bands with similar information into a group, and group centers are selected to obtain a set of different group centers. The different bands in this set come from different groups, and there are significant differences between the different groups. Therefore, the set of different group centers K0 obtained in the first stage also shows significant differences between its different bands. The first stage can be used as a data preprocessing method to achieve the purpose of noise reduction and redundancy removal. In the second stage, the number of selected bands K can be specified, and PCA principal component analysis combined with Shannon's theorem is used to select the first K bands in the first stage set as the final band set.

[0053] For details, see Figure 4As shown, this is a fusion framework structure based on the YOLO target detection model in an embodiment of the present invention. An asymmetric dual-backbone network structure is used to establish the YOLO model. One backbone network is the Backbone network, which selects features from the solid waste RGB dataset, and the other backbone network is the MSI feature extraction network, which extracts features from the multispectral dataset. The features output by the two backbone networks are fused in the PANET unit, and prediction is made based on the fused features.

[0054] For details, see Figure 5 The diagram shown illustrates the training of a dual-backbone YOLO model according to an embodiment of the present invention. This is achieved by truncating the hyperspectral image along its channel directions. Figure 3 The K specified in the diagram represents the band. The RGB image and the cropped multispectral image are simultaneously input into the fusion network for training to obtain the RGB-MSI prediction model.

[0055] For details, see Figure 6 The diagram shown illustrates how an embodiment of the present invention detects solid waste on-site using an RGB-MSI prediction model. The multispectral images and RGB images collected in real time at the solid waste sorting site are simultaneously input into the trained RGB-MSI prediction model, which outputs the coordinates and category of the solid waste to achieve solid waste sorting.

[0056] See Figure 7 As shown, this is an acquisition platform for detecting on-site solid waste using the RGB-MSI prediction model according to an embodiment of the present invention. A Dalsa linear array color camera and a Specim linear array hyperspectral camera (FX17) are installed above the conveyor belt transporting solid waste, and are equipped with LED linear array light sources and dome light sources, respectively. The two cameras acquire images simultaneously, and the acquired RGB and multispectral images are used as inputs to the aforementioned RGB-MSI prediction model.

[0057] See Figure 8 The diagram shown is a system structure diagram of an embodiment of the present invention, which includes the following modules:

[0058] The data acquisition module 801 is used to acquire datasets. It uses a hyperspectral camera and a color camera to acquire hyperspectral images and RGB images of different types of solid waste samples, respectively. N hyperspectral data points are randomly selected from each hyperspectral image. Each data point has M bands, and the data points form an N*M hyperspectral dataset. The RGB images serve as the solid waste RGB dataset.

[0059] The band selection module 802 is used to acquire multispectral feature bands. It transposes the hyperspectral dataset to obtain an M*N transposed hyperspectral dataset, and puts the transposed hyperspectral dataset into the improved AP clustering algorithm to adaptively find the optimal cluster center. The band set K0 of the optimal cluster center is analyzed by PCA method to obtain the spectral bands corresponding to the first K maximum values ​​of the first principal component loading curve of PCA, which are used as multispectral feature bands.

[0060] The dataset construction module 803 is used to acquire a multispectral dataset. Based on the spectral feature bands, the acquired hyperspectral images of solid waste are cropped, and the image channels where the multispectral feature bands are located are cropped to form a pseudo-color image with K channels, which serves as the multispectral dataset.

[0061] Model building module 804 is used to build a dual-backbone YOLO model. It adopts an asymmetric dual-backbone network structure to build the YOLO model. One backbone network extracts features from the solid waste RGB dataset to obtain RGB features; the other backbone network extracts features from the multispectral dataset to obtain MSI features; finally, the features output by the two backbone networks are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and solid waste coordinates.

[0062] Training module 805 is used to obtain the RGB-MSI prediction model. It inputs a multispectral dataset and the corresponding solid waste RGB dataset into a dual-backbone YOLO model. The solid waste RGB dataset is used to train one backbone network to extract features from the solid waste RGB dataset, obtaining RGB features. The multispectral dataset is used to train the other backbone network to extract features from the multispectral dataset, obtaining MSI features. Finally, the obtained RGB and MSI features are fused, and the solid waste category is predicted based on the fused features, outputting the solid waste category and coordinates. The trained dual-backbone YOLO model is then used as the RGB-MSI prediction model.

[0063] Analysis module 806 is used for solid waste detection and sorting. It captures solid waste images using both an on-site RGB camera and a multispectral camera. The two sets of images are simultaneously input into the RGB-MSI prediction model, which outputs the coordinates and category of the solid waste, thus achieving solid waste sorting.

[0064] In this embodiment, RGB and multispectral images of solid waste are collected and input into the RGB-MSI prediction model for solid waste prediction, which can achieve high classification accuracy. Compared with the single input of RGB images, the accuracy of material detection of objects is greatly improved.

[0065] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A solid waste sorting method based on RGB-MSI feature fusion, characterized in that, The method comprises the following steps: S101, a data set acquisition step, using a hyperspectral camera and a color camera to collect hyperspectral images and RGB images of different types of solid waste samples respectively, randomly selecting N hyperspectral data points for each hyperspectral image, each data point having M wavebands, so that the data points form a N*M hyperspectral data set; the RGB image is used as a solid waste RGB data set; S102, a multi-spectral feature waveband acquisition step, transposing the hyperspectral data set to obtain an M*N transposed hyperspectral data set, placing the transposed hyperspectral data set into a modified AP clustering algorithm to adaptively find an optimal clustering center; using a PCA method to analyze a waveband set K0 of the optimal clustering center to obtain spectral wavebands corresponding to the first K maximum values of the PCA first principal component load curve, as multi-spectral feature wavebands; S103, a multi-spectral data set acquisition step, based on the multi-spectral feature wavebands, intercepting the collected solid waste hyperspectral images, intercepting the image channels where the multi-spectral feature wavebands are located, and forming K-channel pseudo-color images as a multi-spectral data set; S104, a double-main-network YOLO model construction step, using an asymmetric double-main-network structure to establish a YOLO model comprising two main networks; one main network is used for feature extraction of the solid waste RGB data set to obtain RGB features; The other main network is used for feature extraction of the multi-spectral data set to obtain MSI features; finally, the features output by the two main networks are fused, the fused features are used for solid waste category prediction, and solid waste coordinates and solid waste categories are output; S105, an RGB-MSI prediction model acquisition step, inputting the multi-spectral data set and the corresponding solid waste RGB data set into the double-main-network YOLO model; the solid waste RGB data set is used to train one of the main networks to extract features of the solid waste RGB data set to obtain RGB features; The multi-spectral data set is used to train the other main network to extract features of the multi-spectral data set to obtain MSI features; Finally, the obtained RGB features and MSI features are fused, the fused features are used for solid waste category prediction, and the output results include solid waste coordinates and solid waste categories; the trained double-main-network YOLO model is used as an RGB-MSI prediction model; S106, a solid waste detection and sorting step, obtaining a solid waste RGB picture by shooting the solid waste with a field RGB camera, obtaining a multi-spectral picture by shooting the solid waste with a multi-spectral camera, inputting the obtained solid waste RGB picture and multi-spectral picture into the RGB-MSI prediction model, and obtaining solid waste coordinates and solid waste categories to realize solid waste sorting.

2. The solid waste sorting method based on RGB-MSI feature fusion according to claim 1, characterized in that, The hyperspectral data points selected in S101 are located in the near-infrared spectral region, and the wavelength is 900nm-1700nm.

3. The solid waste sorting method based on RGB-MSI feature fusion according to claim 2, characterized in that, After the hyperspectral data points in S101 are preprocessed, a hyperspectral data set is formed; the hyperspectral data points have high-dimensional characteristics, and the characteristics are the reflectivity of near-infrared spectra; the preprocessing is black and white frame correction of the reflectivity, and the correction formula is as follows: Wherein, r represents the normalized spectral reflectance, DN is the original spectral data collected, DN w is the white frame data collected using a standard correction white board, DN b is the black frame data collected by covering the hyperspectral camera lens.

4. The solid waste sorting method based on RGB-MSI feature fusion according to claim 1, characterized in that, The improved AP clustering algorithm in S102 adopts the Pearson correlation coefficient to construct a similarity matrix, analyzes the Pearson correlation coefficient of all wave bands two by two, and finally obtains an M*M similarity matrix, which represents the linear relationship between different wave bands. X,Y , Wherein, X, Y represent two variables to be calculated for the Pearson correlation coefficient, sigma X represents the sample standard deviation of X, sigma Y represents the sample standard deviation of Y, and cov represents the covariance between two variables.

5. The solid waste sorting method based on RGB-MSI feature fusion according to claim 4, characterized in that, The preference parameter in the improved AP clustering algorithm in S102 adopts the information entropy formula of Shannon theorem, and the information entropy of the spectral bands is sorted from large to small, and the first N 信 spectral bands are set as the first weight, and the weight of the remaining spectral bands is set as the second weight, wherein the first weight is greater than the second weight; the weight of the spectral band constitutes a 1*M preference array, which represents the preference of different bands to become a clustering center.

6. The solid waste sorting method based on RGB-MSI feature fusion according to claim 5, characterized in that, In S102, the second weight is the median of the similarity matrix.

7. The solid waste sorting method based on RGB-MSI feature fusion according to claim 6, characterized in that, In S102, the center wave band selected by the AP clustering is analyzed by using the PCA analysis, the first principal component of the PCA is selected, the feature vector of the first principal component is output, the feature vector corresponds to the weight value of each wave band, and the first K wave bands with the largest weight value are selected as the final selected multi-spectral feature wave band.

8. The solid waste sorting method based on RGB-MSI feature fusion according to claim 1, characterized in that, In S104, the features output by the two backbone networks are fused, specifically: the RGB features and the MSI features are fused by the concat operation at the rear of the double backbone network.

9. The solid waste sorting method based on RGB-MSI feature fusion according to claim 1, characterized in that, In S105, the training process needs manual or semi-automatic labeling of the data set, and the multi-spectral data set and the RGB data set share one label set.

10. A solid waste sorting system based on RGB-MSI feature fusion, characterized in that, The following modules are included: A data acquisition module is used to obtain a data set, a hyperspectral camera and a color camera are used to collect hyperspectral images and RGB images of different types of solid waste samples, N hyperspectral data points are randomly selected from each hyperspectral image, each data point has M wave bands, and then the data points form an N*M hyperspectral data set; the RGB image is used as a solid waste RGB data set; A wave band selection module is used to obtain a multi-spectral feature wave band, transpose the hyperspectral data set to obtain an M*N transposed hyperspectral data set, and put the transposed hyperspectral data set into the improved AP clustering algorithm to adaptively find the optimal clustering center; the wave band set K0 of the optimal clustering center is analyzed by using the PCA method to obtain the spectral wave band corresponding to the first principal component load curve of the PCA, which is used as the multi-spectral feature wave band; A data set construction module is used to obtain a multi-spectral data set, and according to the spectral feature wave band, the collected solid waste hyperspectral images are intercepted to form K channel pseudo-color images, which are used as the multi-spectral data set; A model construction module is used to construct a double backbone network YOLO model, an asymmetric double backbone network structure is used to establish the YOLO model, one backbone network extracts features from the solid waste RGB data set to obtain RGB features; Another backbone network extracts features from the multi-spectral data set to obtain MSI features; finally, the features output by the two backbone networks are fused, the fused features are used to predict the category of the solid waste, and the solid waste category and the solid waste coordinates are output; A training module is used to obtain an RGB-MSI prediction model, the multi-spectral data set and the corresponding solid waste RGB data set are input into the double backbone network YOLO model, the solid waste RGB data set is used to train one of the backbone networks to extract features from the solid waste RGB data set to obtain RGB features; The multi-spectral data set is used to train another backbone network to extract features from the multi-spectral data set to obtain MSI features; Finally, the obtained RGB features and MSI features are fused, the fused features are used to predict the category of the solid waste, and the solid waste category and the solid waste coordinates are output; the trained double backbone network YOLO model is used as the RGB-MSI prediction model; An analysis module is used for solid waste detection and sorting, and the solid waste is jointly shot by an on-site RGB camera and a multispectral camera, two groups of pictures obtained are simultaneously input into an RGB-MSI prediction model, the model outputs coordinates and categories of the solid waste, and solid waste sorting is realized.

Images