City solid waste fine identification method and system based on RGB-MSI multi-feature fusion

Abstract

The invention discloses an urban solid waste fine identification method and system based on RGB-MSI multi-feature fusion, and the method comprises the steps: collecting RGB data and HSI data of urban solid waste, carrying out the preprocessing through employing a supervised learning-based dual-competition depth Q network, and obtaining a fusion data set composed of the RGB data and MSI data; constructing a multi-scale asymmetric feature fusion network structure based on a Mask R-CNN model, and training by using the fusion data set to obtain an urban solid waste fine recognition model; the urban solid waste on the conveying belt is shot through the on-site RGB camera and the hyperspectral camera together, two sets of pictures with aligned pixels are obtained and input into the industrial personal computer at the same time, the trained urban solid waste fine recognition model is used for prediction, and coordinates and categories of the solid waste are obtained. According to the method, the advantages of a hyperspectral imaging technology and a target detection algorithm are integrated on the basis of a multi-modal fusion method, and the urban solid waste sorting precision is improved by making up for the insufficiency of the hyperspectral imaging technology and the target detection algorithm.

Description

Technical Field

[0001] The present invention relates to the field of hyperspectral imaging technology and multimodal fusion, and in particular to a method and system for finely identifying urban solid waste based on RGB-MSI multi-feature fusion. Background Art

[0002] With the continuous development of deep learning algorithms, intelligent sorting methods based on photoelectric and machine vision are gradually replacing traditional sorting methods such as manual and mechanical sorting. Among them, the photoelectric sorting method based on hyperspectral can use the different spectral characteristics of different types of plastics to quickly and accurately identify the type of plastic, while the target detection and instance segmentation algorithm can obtain the bounding box and label of the instance object for positioning and classification. Therefore, these two methods are widely used in the field of solid waste sorting.

[0003] Hyperspectral camera equipment is expensive, which is not conducive to large-scale industrial use. In addition, HSI contains a lot of redundant information, which requires data dimensionality reduction to obtain MSI. By detecting RGB images of solid waste through the R-CNN series of target detection models, solid waste can be located and classified, and the intelligent sorting of solid waste can be achieved with the actuator. However, as the sorting conditions of urban solid waste become more and more complex, many single-modal target detection models cannot effectively complete the sorting tasks, and relying on manual collection of large data sets to solve the problem of low model generalization also means that higher costs are required, and the accuracy improvement is limited, and it cannot be quickly applied to new scenarios. Therefore, in order to finely identify and sort urban solid waste, there is an urgent need for a multimodal fusion method to combine the advantages of hyperspectral imaging technology and target detection algorithms to make up for the shortcomings of the two and improve the accuracy of urban solid waste sorting. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion, comprising the following steps:

[0006] Collect RGB data and HSI data of urban solid waste to obtain pixel-aligned RGB data sets and HSI data sets; use a dual-competitive deep Q network based on supervised learning to preprocess the HSI data to obtain MSI data. The RGB data set and the MSI data set form a fused data set;

[0007] Based on the Mask R-CNN model, a network structure of multi-scale asymmetric feature fusion is constructed and trained using a fusion data set to obtain a fine urban solid waste recognition model; the fine urban solid waste recognition model uses two branches to extract features from RGB data and MSI data respectively, and in the mid-stage of feature extraction of the two branches, the multi-scale features of RGB and MSI are fused, and finally the target category is predicted based on the fused features;

[0008] The urban solid waste on the conveyor belt is photographed by the on-site RGB camera and the hyperspectral camera, and two sets of pixel-aligned pictures are obtained and input into the industrial computer at the same time. The trained urban solid waste fine recognition model is used for prediction to obtain the coordinates and category of the solid waste.

[0009] Preferably, the method of preprocessing the HSI data using a dual competitive deep Q network based on supervised learning to obtain MSI data comprises the following steps:

[0010] In the HSI dataset, for each category, we randomly select and extract hyperspectral data points from hyperspectral images of different individuals to obtain C categories of hyperspectral data sets, forming a C*N*M preliminary hyperspectral point data set, where C represents the number of categories, N represents the number of samples in each category, and M represents the number of bands in each hyperspectral image.

[0011] The preliminary selected hyperspectral point dataset is input into the double competitive deep Q network based on supervised learning to obtain K spectral bands as MSI feature bands;

[0012] The images in the HSI dataset are intercepted according to the MSI characteristic bands, and the pseudo-color images of the K channel are formed as the MSI dataset.

[0013] Preferably, the HSI data is preprocessed by using a dual competitive deep Q network based on supervised learning to obtain MSI data. The HSI data is first converted into a reflectivity r before being input into the dual competitive deep Q network based on supervised learning, which is expressed as:

[0014]

[0015] DN represents the collected raw HSI data; DN w Represents standard white frame data, obtained by collecting standard white boards; DN b represents the standard black frame data, obtained by covering the camera lens, and r represents the reflectivity data obtained.

[0016] Preferably, the preliminary selected hyperspectral point data set is input into a dual competitive deep Q network based on supervised learning to obtain K spectral bands, specifically: in each action, the agent is designed to filter out a band from the preliminary selected hyperspectral point data set and add it to the current feature band subset; the classification accuracy is fed back to the agent as a reward, and the agent makes a decision to select the next action based on the feedback information and the current state until the termination condition is reached, and finally K spectral bands are obtained as the feature bands of MSI.

[0017] Preferably, the loss function L(w) of the dual competitive deep Q network based on supervised learning is expressed as:

[0018] L(w)=E[(yQ(s,a;w)) 2 ];

[0019]

[0020] Where Q represents the Q value function; s represents the current state, a represents the current action; y represents the learning target value; γ represents the discount factor, w represents the prediction network parameter, and w - is the target network parameter, and r represents the reward.

[0021] Preferably, the K spectral bands finally obtained are used as the characteristic bands of MSI, and the K spectral bands finally obtained are the characteristic band subset with the largest Q value, and the Q value is obtained by adding the state value V of the state and the advantage value A of the action related to the state according to the Q value output in the deep neural network, which is expressed as:

[0022]

[0023] Among them, Q(s,a;w,α,β) represents the sum of the state value function and the action advantage function, which is used to calculate the expected cumulative return of taking actions under the cumulative state, and α and β represent the parameters of the fully connected layer.

[0024] Preferably, the urban solid waste fine identification model includes:

[0025] The feature extraction network extracts RGB features and MSI features through two branches respectively; the RGB image feature extraction branch uses the residual network ResNet50, and the network structure consists of multiple convolutional layers, pooling layers and fully connected layers, and finally outputs a multi-scale RGB feature map; the MSI image feature extraction branch consists of a multi-scale filter group, a multi-spectral image convolution unit and several convolutional layers and pooling layers, and finally outputs a multi-scale MSI feature map;

[0026] Feature fusion, through the concat operation, the RGB feature map and the MSI feature map of the corresponding scale are fused respectively, which is expressed as:

[0027]

[0028] in, Represented as RGB features, Indicates MSI characteristics, P i Represents the fused feature map;

[0029] For target detection, the fused feature map is directly input into the detection head of Mask R-CNN after passing through a 1×1 2D convolution layer to obtain the category and coordinate information of the target object.

[0030] Preferably, the urban solid waste on the conveyor belt is photographed together by an on-site RGB camera and a hyperspectral camera, and the hyperspectral camera collects spectral data of the MSI characteristic band of the urban solid waste.

[0031] The present invention also provides a system for fine identification of urban solid waste based on RGB-MSI multi-feature fusion, comprising:

[0032] The data processing module collects RGB data and HSI data of urban solid waste to obtain pixel-aligned RGB data sets and HSI data sets; a dual-competitive deep Q network based on supervised learning is used to preprocess the HSI data to obtain MSI data. The RGB data and MSI data form a fused data set;

[0033] A model building module, which builds a fine urban solid waste identification model based on the Mask R-CNN model and uses a fusion data set for training; the fine urban solid waste identification model adopts a network structure of multi-scale asymmetric feature fusion, in which one backbone network extracts features from RGB and the other backbone network extracts features from MSI; in the mid-stage of feature extraction, the multi-scale features of RGB and MSI are fused;

[0034] The solid waste identification module uses an on-site RGB camera and a hyperspectral camera to jointly photograph the urban solid waste on the conveyor belt, obtain two sets of pixel-aligned pictures and input them into the industrial computer at the same time, and use the trained urban solid waste fine identification model to make predictions to obtain the coordinates and categories of the solid waste.

[0035] The present invention has the following beneficial effects:

[0036] (1) The HSI band selection method based on deep learning and reinforcement learning is used for the band selection of solid waste HSI. It can automatically select characteristic bands that can effectively distinguish different types of plastic solid waste according to the specified number of characteristic bands, reduce the band redundancy of HSI, and is more suitable for complex and changeable working conditions;

[0037] (2) For complex solid waste sorting conditions, the asymmetric multi-scale feature fusion network can extract features from RGB and MSI respectively through two asymmetric backbone networks, and perform feature fusion on the corresponding color, shape and texture feature information and material spectrum feature information. It makes full use of RGB texture color information and combines it with spectral material information to achieve more accurate identification of solid waste objects; RGB and MSI are pixel-corresponding relationships, so a common label is used without the need for additional labeling.

[0038] (3) The feature fusion idea provides another modality input for the target detection model. The mid-term feature fusion method is used to effectively combine the RGB features and the MSI features, which can effectively increase the detection accuracy of the model. This fusion method is not only applicable to the target detection model, but can also be used in the instance segmentation model and is applicable to other modal image data.

[0039] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A method step diagram of an embodiment of the present invention;

[0041] Figure 2 A solid waste data set sample used for band selection in an embodiment of the present invention;

[0042] Figure 3 A dual competitive deep Q network framework based on supervised band selection according to an embodiment of the present invention;

[0043] Figure 4 It is the framework structure of the urban solid waste fine identification model according to the embodiment of the present invention;

[0044] Figure 5 It is a dual-camera acquisition platform according to an embodiment of the present invention;

[0045] Figure 6 4 is a system structure diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0046] See also Figure 1 FIG. 1 is a diagram showing steps of a method according to an embodiment of the present invention, comprising the following steps:

[0047] S101, collect RGB data and HSI data of urban solid waste to obtain pixel-aligned RGB data sets and HSI data sets; use a dual-competitive deep Q network based on supervised learning to preprocess the HSI data to obtain MSI data, and the RGB data and MSI data form a fused data set;

[0048] S102, constructing a network structure for multi-scale asymmetric feature fusion based on the Mask R-CNN model and training it with the fused data set to obtain a fine urban solid waste recognition model; the fine urban solid waste recognition model uses two branches to extract features from RGB data and MSI data respectively, performs feature fusion on multi-scale features of RGB and MSI in the mid-stage of feature extraction of the two branches, and finally predicts the target category based on the fused features;

[0049] S103, using the on-site RGB camera and the hyperspectral camera to jointly photograph the urban solid waste on the conveyor belt, obtain two sets of pixel-aligned pictures and input them into the industrial computer at the same time, use the trained urban solid waste fine recognition model to make predictions, and obtain the coordinates and categories of the solid waste.

[0050] Specifically, the data collection in S101 is to collect pixel-aligned RGB and HSI image data sets on an urban solid waste RGB-MSI fusion collection platform composed of an RGB area array camera and a hyperspectral camera to obtain a fused data set;

[0051] For details, see Figure 2 As shown in the figure, the original hyperspectral data set has 224 bands in the wavelength range of 935.9nm to 1722.5nm, with a spectral resolution of 3.5nm. The information quality of hyperspectral data is easily affected by the light source and background. In order to eliminate the influence of dark current and the noise caused by the intensity of the light source, it is necessary to use the black and white frame correction method to convert the collected data into reflectance r, which is defined as follows:

[0052]

[0053] Where DN represents the collected raw data, DNw represents the standard white frame data, which is obtained by collecting the standard white board, DNb represents the standard black frame data, which is obtained by covering the camera lens, and r represents the reflectivity data obtained.

[0054] Specifically, the preprocessing in S101 includes: randomly extracting n hyperspectral data points from hyperspectral images of different individuals of the same material type sample in the original hyperspectral data set, forming a total of N samples of the same type khyperspectral data points, and samples of other categories are the same as above, resulting in a C category of urban solid waste, with N samples in each category and M bands in each hyperspectral image, forming a C*N*M original hyperspectral point dataset. The dataset is divided into a training set and a validation set according to a certain ratio and input into a dual competitive deep Q network based on supervised learning; in the dual competitive deep Q network based on supervised learning, in each action, the agent is designed to filter out a band from the original HSI and add it to the current feature band subset; the classification accuracy is fed back to the agent as a reward, and the agent makes a decision to select the next action based on the feedback information and the current state until the termination condition is reached, and finally K spectral bands are obtained as the feature bands of MSI; according to the feature bands of MSI, the collected solid waste hyperspectral images are intercepted, and the image channels where the feature bands of MSI are located are intercepted to form a pseudo-color image of K channels, which is a multispectral dataset (MSI dataset).

[0055] Specifically, S102 modifies the network of the Mask R-CNN model and adopts a network structure of multi-scale asymmetric feature fusion, in which one backbone network extracts features from RGB and the other backbone network extracts features from MSI; in the mid-stage of feature extraction, the multi-scale features of RGB and MSI are fused. The solid waste RGB dataset and the MSI dataset are input into the asymmetric multi-scale feature fusion network to train the fusion network model and finally obtain the Mask R-CNN fusion prediction model, i.e., the urban solid waste fine recognition model.

[0056] For details, see Figure 3 As shown, it is a schematic diagram of a dual competitive deep Q network (DQN) based on supervised learning in an embodiment of the present invention. DQN uses a deep convolutional neural network to approximate the behavior value function Qπ(s,a), and adds two important elements, experience playback and target network, in the iterative process, which improves the learning ability of DQN. At this time, the update of the network training Q function satisfies the formula:

[0057] Q π (s,a)=r+Q π (s',π(s'));

[0058] Where s represents the current state, a represents the current action, and r represents the reward and the next state s'. The left side of the equation is the output of the network, and the right side is the target. During the training process, the weight of the target network needs to be fixed to minimize the error between the model output and the target.

[0059] The loss function between the model output and the target is as follows. After the Q network is updated several times, the target Q network is replaced by the updated Q network:

[0060] L(w)=E[(yQ(s,a;w)) 2 ];

[0061] Where w represents the network weight and y represents the learning target value, as shown below:

[0062]

[0063] Where y represents the target value, γ represents the discount factor, w represents the prediction network parameter, and w represents the target network parameter.

[0064] The output of the Q value in the deep neural network is divided into the state value V (scalar) of the state and the advantage value A (vector) of the action related to the state. Finally, V and A are added to obtain the Q value of each action, as shown in formula (3.6). α and β represent the parameters of the fully connected layer:

[0065]

[0066] Specifically, the image in the HSI data set is intercepted according to the MSI characteristic band, which is to crop the image of the HSI data set in the channel direction, and the MSI data set in the output format of a Tif file is in a one-to-one correspondence with the RGB data set.

[0067] For details, see Figure 4 As shown, the framework structure of the urban solid waste fine identification model of the embodiment of the present invention is shown. The existing Mask R-CNN target detection model is modified to improve the original single-input single-trunk network into a dual-input dual-trunk network. Feature fusion is performed in the middle of feature extraction of the dual-trunk network, and the features of RGB and multi-spectrum are fused through the concat operation. The RGB-MSI multimodal fusion method is used to make up for the shortcomings of insufficient material information in RGB and insufficient color and texture information in MSI. The urban solid waste fine identification model includes:

[0068] The feature extraction network extracts RGB features and MSI features through two branches respectively; each branch consists of a convolutional layer, a pooling layer, and a nonlinear layer, expressed as:

[0069]

[0070] a s =f(z s );

[0071]

[0072] Among them, x irepresents the input RGB data or MSI data; W represents the number of convolution filter groups, q represents the number of convolution kernels; f(·) represents the activation function; s and P represent the space between the pooling unit grids and the size of the spatial region of the pooling unit, respectively;

[0073] Specifically, the RGB image feature extraction branch uses the residual network ResNet50, and the network structure consists of multiple convolutional layers, pooling layers, and fully connected layers. Figure 4 As shown in the figure, the convolution layer of each residual module in ResNet50 is used to extract features; the jump connection avoids the gradient vanishing problem; the pooling layer reduces the dimension of the feature map to reduce the amount of calculation; the nonlinear layer introduces nonlinearity to the model, making the model have stronger fitting ability. At each stage of ResNet50, feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 image sizes are obtained for feature fusion.

[0074] Specifically, the MSI image feature extraction branch consists of a multi-scale filter bank (MFB), a multispectral image convolution unit (MICU), and several convolutional layers and pooling layers. The MFB is an Inception structure composed of 3D convolutional layers with sizes of C×1×1 and C×3×3 and a stride of 1, as shown in Figure 4 As shown in (b), the C×1×1 3D convolutional layer can be used to extract spectral information, and the C×3×3 3D convolutional layer can be used to extract spatial information, where C is the number of multispectral image bands; MICU consists of 1×1 and 3×3 2D convolutional layers, nonlinear layers with ReLU activation function, and maximum pooling layers. The multi-stage output of the MSI image feature extraction branch is the output of each MICU, and feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 image sizes are obtained for feature fusion.

[0075] Feature fusion, through the concat operation, RGB feature maps and MSI feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 image sizes are fused respectively, expressed as:

[0076]

[0077] in, Represented as RGB features, Indicates MSI characteristics;

[0078] For object detection, the concat feature map is directly input into the detection head of Mask R-CNN after passing through a 1×1 2D convolution layer to obtain the category and coordinate information of the target object.

[0079] For details, see Figure 5As shown in the figure, when predicting existing urban solid waste, RGB cameras and multispectral cameras are needed to capture RGB images and multispectral images of objects at the same time. The multispectral camera captures K spectral bands (i.e., the characteristic bands of MSI), and the two sets of pixel-aligned images are synchronously input to the industrial computer. The trained model is used for real-time prediction to obtain the coordinates and categories of the solid waste, thereby improving the accuracy.

[0080] For details, see Figure 6 FIG. 1 is a system structure diagram of an embodiment of the present invention, including:

[0081] The data processing module 601 collects RGB data and HSI data of urban solid waste to obtain pixel-aligned RGB data sets and HSI data sets; a dual-competitive deep Q network based on supervised learning is used to pre-process the HSI data to obtain MSI data, and the RGB data and the MSI data form a fused data set;

[0082] Model building module 602, building a fine urban solid waste identification model based on the Mask R-CNN model and using the fused data set for training; the fine urban solid waste identification model adopts a network structure of multi-scale asymmetric feature fusion, in which one backbone network extracts features from RGB and the other backbone network extracts features from MSI; in the mid-stage of feature extraction, the multi-scale features of RGB and MSI are fused;

[0083] The solid waste identification module 603 uses the on-site RGB camera and the hyperspectral camera to jointly shoot the urban solid waste on the conveyor belt, obtains two sets of pixel-aligned pictures and inputs them into the industrial computer at the same time, and uses the trained urban solid waste fine identification model to make predictions to obtain the coordinates and categories of the solid waste.

[0084] The present invention combines the advantages of hyperspectral imaging technology and target detection algorithm based on a multimodal fusion method, and improves the accuracy of urban solid waste sorting by making up for their deficiencies.

[0085] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion, characterized in that: The following steps are involved: Collect RGB data and HSI data of urban solid waste to obtain pixel-aligned RGB data sets and HSI data sets; A dual competitive deep Q network based on supervised learning is used to preprocess the HSI data to obtain the MSI data. The RGB dataset and the MSI dataset form a fusion dataset. Based on the Mask R-CNN model, a network structure of multi-scale asymmetric feature fusion is constructed and trained using a fusion data set to obtain a fine urban solid waste recognition model; the fine urban solid waste recognition model uses two branches to extract features from RGB data and MSI data respectively, and in the mid-stage of feature extraction of the two branches, the multi-scale features of RGB and MSI are fused, and finally the target category is predicted based on the fused features; The urban solid waste on the conveyor belt is photographed by the on-site RGB camera and the hyperspectral camera, and two sets of pixel-aligned pictures are obtained and input into the industrial computer at the same time. The trained urban solid waste fine recognition model is used for prediction to obtain the coordinates and category of the solid waste.

2. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 1 is characterized in that: The method of preprocessing HSI data using a dual competitive deep Q network based on supervised learning to obtain MSI data includes the following steps: In the HSI dataset, for each category, we randomly select and extract hyperspectral data points from hyperspectral images of different individuals to obtain C categories of hyperspectral data sets, forming a C*N*M preliminary hyperspectral point data set, where C represents the number of categories, N represents the number of samples in each category, and M represents the number of bands in each hyperspectral image. The preliminary selected hyperspectral point dataset is input into the double competitive deep Q network based on supervised learning to obtain K spectral bands as MSI feature bands; The images in the HSI dataset are intercepted according to the MSI characteristic bands, and the pseudo-color images of the K channel are formed as the MSI dataset.

3. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 1 is characterized in that: The dual competitive deep Q network based on supervised learning is used to preprocess the HSI data to obtain MSI data. The HSI data is first converted into reflectivity r before being input into the dual competitive deep Q network based on supervised learning, which is expressed as: DN represents the collected raw HSI data; DN w Represents standard white frame data, obtained by collecting standard white boards; DN b represents the standard black frame data, obtained by covering the camera lens, and r represents the reflectivity data obtained.

4. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 2 is characterized in that: The preliminary selected hyperspectral point data set is input into a double competitive deep Q network based on supervised learning to obtain K spectral bands, specifically: in each action, the intelligent agent is designed to filter out a band from the preliminary selected hyperspectral point data set and add it to the current feature band subset; The classification accuracy is fed back to the agent as a reward, and the agent makes a decision to select the next action based on the feedback information and the current state until the termination condition is reached, and finally K spectral bands are obtained as the characteristic bands of MSI.

5. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 1 is characterized in that: The loss function L(w) of the dual competitive deep Q network based on supervised learning is expressed as: L(w)=E[(y-Q(s,a;w)) 2 ]; Where Q represents the Q value function; s represents the current state, a represents the current action; y represents the learning target value; γ represents the discount factor, w represents the prediction network parameter, and w - is the target network parameter, and r represents the reward.

6. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 1 is characterized in that: The K spectral bands finally obtained are used as the characteristic bands of MSI. The K spectral bands finally obtained are the characteristic band subset with the largest Q value. The Q value is obtained by adding the state value V of the state and the advantage value A of the action related to the state according to the Q value output in the deep neural network, which is expressed as: Among them, Q(s,a;w,α,β) represents the sum of the state value function and the action advantage function, which is used to calculate the expected cumulative return of taking actions under the cumulative state, and α and β represent the parameters of the fully connected layer.

7. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 1 is characterized in that: The urban solid waste fine identification model includes: The feature extraction network extracts RGB features and MSI features through two branches respectively; the RGB image feature extraction branch uses the residual network ResNet50, and the network structure consists of multiple convolutional layers, pooling layers and fully connected layers, and finally outputs a multi-scale RGB feature map; the MSI image feature extraction branch consists of a multi-scale filter group, a multi-spectral image convolution unit and several convolutional layers and pooling layers, and finally outputs a multi-scale MSI feature map; Feature fusion, through the concat operation, the RGB feature map and the MSI feature map of the corresponding scale are fused respectively, which is expressed as: in, Represented as RGB features, Indicates MSI characteristics, P i Represents the fused feature map; For target detection, the fused feature map is directly input into the detection head of Mask R-CNN after passing through a 1×1 2D convolution layer to obtain the category and coordinate information of the target object.

8. The method for fine identification of urban solid waste based on RGB-MSI multi-feature fusion according to claim 2 is characterized in that: The urban solid waste on the conveyor belt is photographed together by an on-site RGB camera and a hyperspectral camera, and the hyperspectral camera collects spectral data of the MSI characteristic band of the urban solid waste.

9. A system for fine identification of urban solid waste based on RGB-MSI multi-feature fusion, characterized in that: include: The data processing module collects RGB data and HSI data of urban solid waste and obtains pixel-aligned RGB data sets and HSI data sets; A dual competitive deep Q network based on supervised learning is used to preprocess the HSI data to obtain MSI data. The RGB data and MSI data form a fusion data set. A model building module, which builds a fine urban solid waste recognition model based on the Mask R-CNN model and uses a fusion data set for training; the fine urban solid waste recognition model adopts a network structure of multi-scale asymmetric feature fusion, in which one backbone network extracts features from RGB and the other backbone network extracts features from MSI; In the mid-stage of feature extraction, the multi-scale features of RGB and MSI are fused; The solid waste identification module uses an on-site RGB camera and a hyperspectral camera to jointly photograph the urban solid waste on the conveyor belt, obtain two sets of pixel-aligned pictures and input them into the industrial computer at the same time, and use the trained urban solid waste fine identification model to make predictions to obtain the coordinates and categories of the solid waste.

Images